Go to hazelcast.org/documentation

Clustered JMX

Clustered JMX via Management Center will allow you to monitor clustered statistics of distributed objects from a JMX interface.

Clustered JMX Configuration

In order to configure Clustered JMX, you should use two command line parameters for your Management Center deployment.

  • -Dhazelcast.mc.jmx.enabled=true (default is false)
  • -Dhazelcast.mc.jmx.port=9000 (optional, default is 9999)

With embedded Jetty, there is no need to deploy your Management Center application to any container or application server.

You can start Management Center application with Clustered JMX enabled as shown below.

java -Dhazelcast.mc.jmx.enabled=true -Dhazelcast.mc.jmx.port=9999 -jar mancenter-3.3.jar

Once Management Center starts, you should see a log similar to below.

INFO: Management Center 3.3
Jun 05, 2014 11:55:32 AM com.hazelcast.webmonitor.service.jmx.impl.JMXService
INFO: Starting Management Center JMX Service on port :9999

You should be able to connect to Clustered JMX interface from the address localhost:9999.

You can use jconsole or any other JMX client to monitor your Hazelcast Cluster. As a sample, below is the jconsole screenshot of the Clustered JMX hierarchy.

API Documentation

The management beans are exposed with the following object name format.

ManagementCenter[cluster name]:type=<object type>,name=<object name>,member="<cluster member IP address>"

Object name starts with ManagementCenter prefix. Then it has the cluster name in brackets followed by a colon. After that, type,name and member attributes follows, each separated with a comma.

  • type is the type of object. Values are Clients, Executors, Maps, Members, MultiMaps, Queues, Services, and Topics.

  • name is the name of object.

  • member is the node address of object (only required if the statistics are local to the node).

A sample bean is shown below.

ManagementCenter[dev]:type=Services,name=OperationService,member="192.168.2.79:5701"

Below is the list of attributes that are exposed from the Clustered JMX interface.

  • ManagementCenter[ClusterName]
  • Clients
    • Address
    • ClientType
    • Uuid
  • Executors
    • Cluster
    • Name
    • StartedTaskCount
    • CompletedTaskCount
    • CancelledTaskCount
    • PendingTaskCount
  • Maps
    • Cluster
    • Name
    • BackupEntryCount
    • BackupEntryMemoryCost
    • CreationTime
    • DirtyEntryCount
    • Events
    • GetOperationCount
    • HeapCost
    • Hits
    • LastAccessTime
    • LastUpdateTime
    • LockedEntryCount
    • MaxGetLatency
    • MaxPutLatency
    • MaxRemoveLatency
    • OtherOperationCount
    • OwnedEntryCount
    • PutOperationCount
    • RemoveOperationCount
  • Members
    • ConnectedClientCount
    • HeapFreeMemory
    • HeapMaxMemory
    • HeapTotalMemory
    • HeapUsedMemory
    • IsMaster
    • OwnedPartitionCount
  • MultiMaps
    • Cluster
    • Name
    • BackupEntryCount
    • BackupEntryMemoryCost
    • CreationTime
    • DirtyEntryCount
    • Events
    • GetOperationCount
    • HeapCost
    • Hits
    • LastAccessTime
    • LastUpdateTime
    • LockedEntryCount
    • MaxGetLatency
    • MaxPutLatency
    • MaxRemoveLatency
    • OtherOperationCount
    • OwnedEntryCount
    • PutOperationCount
    • RemoveOperationCount
  • Queues
    • Cluster
    • Name
    • MinAge
    • MaxAge
    • AvgAge
    • OwnedItemCount
    • BackupItemCount
    • OfferOperationCount
    • OtherOperationsCount
    • PollOperationCount
    • RejectedOfferOperationCount
    • EmptyPollOperationCount
    • EventOperationCount
    • CreationTime
  • Services
    • ConnectionManager
      • ActiveConnectionCount
      • ClientConnectionCount
      • ConnectionCount
    • EventService
      • EventQueueCapacity
      • EventQueueSize
      • EventThreadCount
    • OperationService
      • ExecutedOperationCount
      • OperationExecutorQueueSize
      • OperationThreadCount
      • RemoteOperationCount
      • ResponseQueueSize
      • RunningOperationsCount
    • PartitionService
      • ActivePartitionCount
      • PartitionCount
    • ProxyService
      • ProxyCount
    • ManagedExecutor[hz::async]
      • Name
      • CompletedTaskCount
      • MaximumPoolSize
      • PoolSize
      • QueueSize
      • RemainingQueueCapacity
      • Terminated
    • ManagedExecutor[hz::client]
      • Name
      • CompletedTaskCount
      • MaximumPoolSize
      • PoolSize
      • QueueSize
      • RemainingQueueCapacity
      • Terminated
    • ManagedExecutor[hz::global-operation]
      • Name
      • CompletedTaskCount
      • MaximumPoolSize
      • PoolSize
      • QueueSize
      • RemainingQueueCapacity
      • Terminated
    • ManagedExecutor[hz::io]
      • Name
      • CompletedTaskCount
      • MaximumPoolSize
      • PoolSize
      • QueueSize
      • RemainingQueueCapacity
      • Terminated
    • ManagedExecutor[hz::query]
      • Name
      • CompletedTaskCount
      • MaximumPoolSize
      • PoolSize
      • QueueSize
      • RemainingQueueCapacity
      • Terminated
    • ManagedExecutor[hz::scheduled]
      • Name
      • CompletedTaskCount
      • MaximumPoolSize
      • PoolSize
      • QueueSize
      • RemainingQueueCapacity
      • Terminated
    • ManagedExecutor[hz::system]
      • Name
      • CompletedTaskCount
      • MaximumPoolSize
      • PoolSize
      • QueueSize
      • RemainingQueueCapacity
      • Terminated
  • Topics
    • Cluster
    • Name
    • CreationTime
    • PublishOperationCount
    • ReceiveOperationCount

Enterprise Integration

You can use Clustered JMX interface to integrate Hazelcast Management Center with your own monitoring tool or enterprise solutions like New Relic or AppDynamics.

New Relic Integration

In order to integrate Clustered JMX with New Relic, you need to attach New Relic Java agent and provide an extension file that describes which metrics will be sent to New Relic.

Please see Custom JMX instrumentation by YAML on New Relic webpage.

Below is an example Map monitoring .yml file for New Relic.

name: Clustered JMX
version: 1.0
enabled: true

jmx:
  - object_name: ManagementCenter[clustername]:type=Maps,name=mapname
    metrics:
      - attributes: PutOperationCount, GetOperationCount, RemoveOperationCount, Hits,\ 
            BackupEntryCount, OwnedEntryCount, LastAccessTime, LastUpdateTime
        type: simple
  - object_name: ManagementCenter[clustername]:type=Members,name="node address in\
        double quotes"
    metrics:
      - attributes: OwnedPartitionCount
        type: simple

You should put the .yml file under the extensions folder in your New Relic installation. If that folder does not exist, please create one.

After you set your extension, you can attach New Relic Java agent and start Management Center as shown below.

java -javaagent:/path/to/newrelic.jar -Dhazelcast.mc.jmx.enabled=true\
    -Dhazelcast.mc.jmx.port=9999 -jar mancenter-3.3.jar

Once started and if your logging level is set as FINER, you should see the logs similar to below in the file newrelic_agent.log, which is located at logs folder in your New Relic installation.

Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINE:
    JMX Service : querying MBeans (1)
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER:
    JMX Service : MBeans query ManagementCenter[dev]:type=Members,
    name="192.168.2.79:5701", matches 1
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER:
    Recording JMX metric OwnedPartitionCount : 68
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER:
    JMX Service : MBeans query ManagementCenter[dev]:type=Maps,name=orders, 
    matches 1
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER: 
    Recording JMX metric Hits : 46,593
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER: 
    Recording JMX metric BackupEntryCount : 1,100
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER: 
    Recording JMX metric OwnedEntryCount : 1,100
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER: 
    Recording JMX metric RemoveOperationCount : 0
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER: 
    Recording JMX metric PutOperationCount : 118,962
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER: 
    Recording JMX metric GetOperationCount : 0
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER: 
    Recording JMX metric LastUpdateTime : 1,401,962,426,811
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER: 
    Recording JMX metric LastAccessTime : 1,401,962,426,811

Then, you can navigate to your New Relic account and create Custom Dashboards. Please see Creating custom dashboards on creating one.

While you are creating the dashboard, you should see the metrics that you are sending to New Relic from Management Center in the Metrics section under the JMX folder.

AppDynamics Integration

In order to integrate Clustered JMX with the AppDynamics, you need to attach AppDynamics Java agent to the Management Center.

For agent installation, you can refer to Install the App Agent for Java page.

For monitoring on AppDynamics, please see Using AppDynamics for JMX Monitoring page.

After installing AppDynamics agent, you can start Management Center as shown below.

java -javaagent:/path/to/javaagent.jar -Dhazelcast.mc.jmx.enabled=true\
    -Dhazelcast.mc.jmx.port=9999 -jar mancenter-3.3.jar

Once started, you should see the below logs.

Started AppDynamics Java Agent Successfully.
Hazelcast Management Center starting on port 8080 at path : /mancenter