This manual is for an old version of Hazelcast Management Center, use the latest stable version.
This manual is for an old version of Hazelcast Management Center, use the latest stable version.

Welcome to the Reference Manual of Hazelcast IMDG Management Center. This manual includes concepts and instructions to guide you on how to use Management Center to monitor your Hazelcast IMDG Cluster.

Preface

Hazelcast Management Center enables you to monitor and manage your cluster members running Hazelcast IMDG. In addition to monitoring the overall state of your clusters, you can also analyze and browse your data structures in detail, update map configurations and take thread dumps from the members. You can run scripts (JavaScript, Groovy, etc.) and commands on your members with its scripting and console modules.

Naming

  • Hazelcast Management Center or Management Center refers to the Hazelcast IMDG cluster monitoring tool provided by Hazelcast, Inc.

  • Hazelcast IMDG or just Hazelcast refers to the Hazelcast in-memory data grid middleware. Hazelcast is also the name of the company (Hazelcast, Inc.) providing Hazelcast IMDG.

Hazelcast IMDG

Hazelcast Management Center is delivered with Hazelcast IMDG. It can also be downloaded as a separate package from the hazelcast.org website.

See the Hazelcast IMDG Reference Manual for all Hazelcast IMDG topics including the clusters and their operations, clients, data structures, computing and WAN replication.

Licensing

This Reference Manual is free and provided under the Apache License, Version 2.0.

Hazelcast Management Center requires either a Management Center license or Hazelcast IMDG Enterprise license or Hazelcast IMDG Enterprise HD license. It also has a free version which lets you to monitor your cluster having up to three IMDG members.

Trademarks

Hazelcast is a registered trademark of Hazelcast, Inc. All other trademarks in this manual are held by their respective owners.

Getting Help

Support is provided via the following channels:

Release Notes

See the Release Notes document for the new features, enhancements and fixes performed for each Hazelcast Management Center release.

1. Browser Compatibility

The Hazelcast Management Center is tested and works on the following browsers:

  • Last two versions of Chrome at the time of release

  • Last two versions of Firefox at the time of release

  • Latest Safari

  • Internet Explorer 11

  • Latest Microsoft Edge

2. Getting Started

To start using the Management Center:

  1. download the Hazelcast IMDG or Management Center package

  2. start the Management Center service

  3. launch the Management Center user interface.

2.1. Downloading Management Center

Hazelcast Management Center is included in the Hazelcast IMDG download package. You can download it from the download page of Hazelcast’s website.

When a new Hazelcast IMDG version is released, it comes with a Management Center that is compatible with IMDG. There may be times when a new Management Center version is released before a new version of Hazelcast IMDG. In that case, you may download the new Management Center from its download page as a separate package.

2.2. Starting the Management Center Service

You have the following options to start the Management Center service:

  • starting Hazelcast Management Center from the command line

  • using the scripts that come with the download package.

  • deploying the file hazelcast-management-center-4.0.war on your Java application server/container

You need Java Runtime Environment 1.8 or later to run Hazelcast Management Center.

2.2.1. Using the Command Line

After you downloaded, extract the Hazelcast IMDG or Management Center package. The extracted directory, i.e., hazelcast-management-center-4.0, contains the hazelcast-management-center-4.0.war file.

You can start this file directly from the command line using the following command:

java -jar hazelcast-management-center-4.0.war

The above command starts the Hazelcast Management Center service on the port 8080 with the root context path (http://localhost:8080).

In Management Center 4.0, the default context path changed from hazelcast-mancenter to the root context path.

You can use -Dhazelcast.mc.http.port and -Dhazelcast.mc.contextPath command line options to start Management Center on a different port and context path:

java -Dhazelcast.mc.http.port=8083 -Dhazelcast.mc.contextPath='hazelcast-mc' -jar hazelcast-management-center-4.0.war

The above command starts Management Center on port 8083 and context path hazelcast-mc. You can access it by opening http://localhost:8083/hazelcast-mc in your browser.

Then, you need to let Management Center know the Hazelcast IMDG member addresses as explained in the Connecting IMDG Members to Management Center chapter.

For the options you can provide when starting with the command line, see the Configuring Management Center chapter. In that chapter, you can learn about topics including how to start with a license or extra classpath, how to configure the security, disk usage, update interval or logging.

2.2.2. Deploying to Application Server

Instead of starting at the command line, you can deploy the Management Center to your application server (Tomcat, Jetty, etc.).

2.2.3. Using Scripts in the Package

As another option, you can use the start.bat or start.sh scripts, which come with the download package, to start the Management Center. You can find these scripts under the extracted directory.

2.3. Quick Launch

After the above steps are completed, let’s quickly open the Management Center user interface with the default settings.

Assuming that you have started your Hazelcast IMDG cluster with a single member on your local machine and the Management Center service on http://localhost:8080, follow the below instructions:

  1. Go to http://localhost:8080 using your preferred browser.

  2. Create your credentials in the "Configure Security" dialog:

    1. Select Default as the security provider.

    2. Create your username and password.

    3. Click on the Save button.

  3. Login with your newly created credentials, and "Manage Clusters" page appears.

  4. Click on the "Add Cluster Config" button to connect Management Center to your cluster.

  5. Click on the Save button with the default values (dev and localhost) on the "Add Cluster Config" page.

  6. Click on the Select button on the "Manage Clusters" page and the dashboard of your cluster appears as shown below.

Dashboard

Also see here in the Hazelcast IMDG Reference Manual for information on starting a Hazelcast member.

3. Configuring Management Center

This chapter explains how you can configure the Management Center according to your needs.

The way Management Center connects and communicates with IMDG clusters has been redesigned in Management Center 4.0

From Management Center 4.0 and onwards, the communications between the IMDG cluster members and the Management Center instance are based on the IMDG client protocol. The IMDG clusters that Management Center should connect to are now configured within Management Center as described in Connecting Management Center to IMDG Members.

In previous versions, the URL of the Management Center instance was configured within the <hazelcast></hazelcast> section of hazelcast.xml for each member of the IMDG cluster. This configuration is now depreciated and ignored.

3.1. Providing a License

When starting the Management Center from the command line, a license can be provided using the system property hazelcast.mc.license. For example by using the command line parameter:

java -Dhazelcast.mc.license=<key> -jar hazelcast-management-center-4.0.war

When this option is used, the license provided takes precedence over any license set and stored previously using the user interface. Previously stored licenses are not affected and will be used again when the Management Center is started without the hazelcast.mc.license property. This also means no new license can be stored when the property is used.

3.2. Providing an Extra Classpath

You can also start the Management Center with an extra classpath entry (for example, when using JAAS authentication) by using the following command:

java -cp "hazelcast-management-center-4.0.war:/path/to/an/extra.jar" com.hazelcast.webmonitor.Launcher

On Windows, the command becomes as follows (semicolon instead of colon):

java -cp "hazelcast-management-center-4.0.war;/path/to/an/extra.jar" com.hazelcast.webmonitor.Launcher

3.3. Configuring the Client Used by Management Center

You can configure the client instance that is used for connecting to the cluster by using the following command line parameters:

  • -Dhazelcast.mc.client.initial.backoff.millis: Duration, in milliseconds, to wait after the first failure before retrying. It is in milliseconds. The default value is 1000 ms. Set values have to be in range of 1000 to 60000 ms.

  • -Dhazelcast.mc.client.backoff.multiplier: Factor with which to multiply backoff after a failed retry. Default value is 2. Set values have to be in range of 1 to 10.

  • -Dhazelcast.mc.client.max.backoff.millis: When backoff reaches this upper bound, it does not increase any more. It is in milliseconds. Default value is 32000 ms. Set values have to be in range of 32000 to 600000 ms.

Note that these parameters apply to all clients that are started by Management Center.

You can also pass in a custom client configuration file to Management Center. These parameters override any configuration you pass as part of adding a new cluster connection. Following is a list of all other client configuration parameters that are overriden by Management Center when creating clients to connect to clusters:

  • InstanceName: Generated based on the cluster name.

  • ConnectionStrategyConfig.asyncStart: Set to true. It means that the client starts without waiting to get a cluster connection.

  • ConnectionStrategyConfig.clusterConnectTimeoutMillis: Set to infinity. The client never gives up trying to connect to the cluster.

  • NetworkConfig.smartRouting: Set to true. It means that client will open connections to all members.

3.4. Configuring Disk Usage

The disk space used by the Management Center is constrained to avoid exceeding available disk space. When the set limit is exceeded, the Management Center deals with this in the following ways:

  • Persisted statistics data is removed, starting with the oldest (one month at a time).

Usually, either of the above automatically resolves the situation and makes room for new data. Depending on the disk usage configuration and the kind of data that contributes to exceeding the limit it can occur that the limit continues to be exceeded. In this case, the Management Center does not store new metrics data. Other data (like configurations and account information) is still stored as they hardly cause larger data volumes.

An active blockage is reported in the UI as an error notification, as shown below:

Disk Usage Limit Error

However, storage operations do not explicitly fail or report errors since this would constantly cause interruptions and error logging both in the UI and logs.

One way to resolve a blockage is to restart the Management Center with a higher limit or in the purge mode (if not used before).

You can use the following system properties to configure the Management Center’s disk usage control:

  • -Dhazelcast.mc.disk.usage.mode: Available values are purge and block. If the mode is purge, persisted statistics data is removed (as stated in the beginning of this section). If it is block, persisted statistics data is not removed. Its default value is purge.

  • -Dhazelcast.mc.disk.usage.limit: The high water mark in KB, MB or GB. Its default value adapts to the available disk space and the space already used by database files. At a maximum it will default to 512MB unless existing data already exceeds this maximum. In that case the already used space is used as limit. The minimal allowed limit is 2MB.

  • -Dhazelcast.mc.disk.usage.interval: Specifies how often the disk usage is checked to see if it exceeds the limit (hazelcast.mc.disk.usage.limit). It is in milliseconds and its default value is 1000 milliseconds. Set values have to be in range of 50 to 5000 ms.

It is important to understand that the limit given is a soft limit, a high water mark. The Management Center will act if it is exceeded but it might be exceeded by a margin between two measurements. Do not set it to the absolute maximum disk space available. A smaller interval increases accuracy but also performance overhead.

In case of a misconfiguration of any of the above three properties, the Management Center logs the problem and aborts startup immediately.

3.5. Enabling Health Check Endpoint

When running the Management Center from the command line, you can enable the Health Check endpoint. This endpoint responds with 200 OK HTTP status code once the Management Center web application has started. The endpoint is available on port <Management Center HTTP port> + 1 with context path <Management Center context path>/health (by default, its URL is http://localhost:8081/health). Note that the HTTP protocol is always used for the Health Check endpoint, independently of TLS/SSL settings, and no additional authentication is enforced for it.

If you want to enable the Health Check endpoint, use the following command line argument:

-Dhazelcast.mc.healthCheck.enable=true

3.6. Configuring Sessions

This section provides information on how to configure the Management Center sessions for various aspects including timeouts and login/logout operations.

3.6.1. Configuring Session Timeout

If you have started the Management Center from the command line by using the WAR file, by default, the sessions that are inactive for 30 minutes are invalidated. To change this, you can use the -Dhazelcast.mc.session.timeout.seconds command line parameter.

For example, the following command starts the Management Center with a session timeout period of 1 minute:

java -Dhazelcast.mc.session.timeout.seconds=60 -jar hazelcast-management-center-4.0.war

If you have deployed the Management Center on an application server/container, you can configure the default session timeout period of the application server/container to change the session timeout period for the Management Center. If your server/container allows application specific configuration, you can use it to configure the session timeout period for the Management Center.

3.6.2. Disabling Multiple Simultaneous Login Attempts

Normally, a user account on the Management Center can be used from multiple locations at the same time. If you want to forbid others from logging in, when there’s already someone logged in with the same username, you can start the Management Center with the -Dhazelcast.mc.allowMultipleLogin=false command line parameter.

3.6.3. Disable Login Configuration

In order to prevent password guessing attacks, logging in is disabled temporarily after a number of failed login attempts. When not configured explicitly, the default values are used, i.e., logging in is disabled for 5 seconds when a username is failed to log in consecutively 3 times. During this 5 seconds of period, logging in is not allowed even when the correct credentials are used. After 5 seconds, the user will be able to log in using the correct credentials.

Assuming the configuration with the default values, if the failed attempts continue (consecutively 3 times) after the period of disabled login passes, this time the disable period is multiplied by 10: logging in is disabled for 50 seconds. The whole process repeats itself until the user logs in successfully. By default, there’s no upper limit to the disable period, but can be configured by using the -Dhazelcast.mc.maxDisableLoginPeriod parameter.

Here is a scenario, in the given order, with the default values:

  1. You try to login with your credentials consecutively 3 times but failed.

  2. Logging in is disabled and you have to wait for 5 seconds.

  3. After 5 seconds have passed, logging in is enabled.

  4. You try to login with your credentials consecutively 3 times but again failed.

  5. Logging in is disabled again and this time you have to wait for 50 seconds until your next login attempt.

  6. And so on; each 3 consecutive login failures causes the disable period to be multiplied by 10.

You can configure the number of failed login attempts, initial and maximum duration of the disabled login and the multiplier using the following command line parameters:

  • -Dhazelcast.mc.failedAttemptsBeforeDisableLogin: Number of failed login attempts that cause the logging in to be disabled temporarily. Default value is 3.

  • -Dhazelcast.mc.initialDisableLoginPeriod: Initial duration for the disabled login in seconds. Default value is 5.

  • -Dhazelcast.mc.disableLoginPeriodMultiplier: Multiplier used for extending the disable period in case the failed login attempts continue after disable period passes. Default value is 10.

  • -Dhazelcast.mc.maxDisableLoginPeriod: Maximum amount of time for the disable login period. This parameter does not have a default value. By default, disabled login period is not limited.

3.6.4. Forcing Logout on Multiple Simultaneous Login Attempts

If you haven’t allowed multiple simultaneous login attempts explicitly, the first user to login with a username stays logged in until that username explicitly logs out or its session expires. In the meantime, no one else can login with the same username. If you want to force logout for the first user and let the newcomer login, you need to start Management Center with the -Dhazelcast.mc.forceLogoutOnMultipleLogin=true command line parameter.

3.7. Configuring and Enabling Security

This section provides information on how to use and manage the Management Center with TLS/SSL and mutual authentication. You will also learn how to force the users to specify passwords that are hard to guess.

3.7.1. Using Management Center with TLS/SSL Only

To encrypt data transmitted over all channels of the Management Center using TLS/SSL, make sure you do all of the following:

  • Deploy the Management Center on a TLS/SSL enabled container or start it from the command line with TLS/SSL enabled. See Installing the Management Center.

    • Another option is to place the Management Center behind a TLS-enabled reverse proxy. In that case, make sure your reverse proxy sets the necessary HTTP header (X-Forwarded-Proto) for resolving the correct protocol.

  • Enable TLS/SSL communication to the Management Center for your Hazelcast cluster. See Connecting Hazelcast members to the Management Center.

  • If you’re using Clustered JMX on the Management center, enable TLS/SSL for it. See Enabling TLS/SSL for Clustered JMX.

  • If you’re using LDAP authentication, make sure you use LDAPS or enable the "Start TLS" field. See LDAP Authentication.

3.7.2. Enabling TLS/SSL When Starting with WAR File

When you start the Management Center from the command line, it serves the pages unencrypted by using "http", by default. To enable TLS/SSL, use the following command line parameters when starting the Management Center:

  • -Dhazelcast.mc.tls.enabled: Specifies whether TLS/SSL is enabled. Its default value is false (disabled).

  • -Dhazelcast.mc.tls.keyStore: Path to the keystore.

  • -Dhazelcast.mc.tls.keyStorePassword: Password of the keystore.

  • -Dhazelcast.mc.tls.trustStore: Path to the truststore.

  • -Dhazelcast.mc.tls.trustStorePassword: Password of the truststore.

You can leave the truststore and truststore password values empty to use the system JVM’s own truststore.

The following is an example on how to start the Management Center with TLS/SSL enabled from the command line:

java -Dhazelcast.mc.tls.enabled=true
-Dhazelcast.mc.tls.keyStore=/some/dir/selfsigned.jks
-Dhazelcast.mc.tls.keyStorePassword=yourpassword -jar hazelcast-management-center-4.0.war

You can access the Management Center from the following HTTPS URL on port 8443: https://localhost:8443.

On the member side, you need to configure the Management Center URL as https://localhost:8443 and also set the following JVM arguments when starting the member:

-Djavax.net.ssl.trustStore=path to your truststore
-Djavax.net.ssl.trustStorePassword=yourpassword
If you plan to use a self-signed certificate, make sure to create a certificate with the hostname of the machine you will deploy the Management Center on. Otherwise, you will see a line similar to the following in the member logs:
javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException:
No subject alternative names matching IP address 127.0.0.1 found

To override the HTTPS port, you can use the -Dhazelcast.mc.https.port command line option when starting the Management Center. For example:

java -Dhazelcast.mc.tls.enabled=true \
     -Dhazelcast.mc.tls.keyStore=/dir/to/certificate.jks \
     -Dhazelcast.mc.tls.keyStorePassword=yourpassword \
     -Dhazelcast.mc.https.port=443 \
     -jar hazelcast-management-center-4.0.war

This starts the Management Center on HTTPS port 443.

You can encrypt the keystore/truststore passwords and pass them as command line arguments in encrypted form for improved security. See the Variable Replacers section for more information.
Enabling HTTP Port

By default, HTTP port is disabled when you enable TLS. If you want to have an open HTTP port that redirects to the HTTPS port, use the following command line argument:

-Dhazelcast.mc.tls.enableHttpPort=true
Managing TLS Enabled Clusters

If a Hazelcast cluster is configured to use TLS for communication between its members using a self-signed certificate, the Management Center will not be able to perform some of the operations that use the cluster’s HTTP endpoints (such as shutting down a member or getting the thread dump of a member). This is so because self-signed certificates are not trusted by default by the JVM. For these operations to work, you need to configure a truststore containing the public key of the self-signed certificate when starting the JVM of the Management Center using the following command line parameters:

  • -Dhazelcast.mc.httpClient.tls.trustStore: Path to the truststore.

  • -Dhazelcast.mc.httpClient.tls.trustStorePassword: Password of the truststore.

  • -Dhazelcast.mc.httpClient.tls.trustStoreType: Type of the truststore. Its default value is JKS.

  • -Dhazelcast.mc.httpClient.tls.trustManagerAlgorithm: Name of the algorithm based on which the authentication keys are provided. System default is used if none is provided. You can find out the default by calling the javax.net.ssl.TrustManagerFactory#getDefaultAlgorithm method.

You can encrypt the truststore password and pass it as a command line argument in encrypted form for improved security. See the Variable Replacers section for more information.

By default, JVM also checks for the validity of the hostname of the certificate. If this test fails, you will see a line similar to the following in the Management Center logs:

javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException:
No subject alternative names matching IP address 127.0.0.1 found

If you want to disable this check, start the Management Center with the following command line parameter:

-Dhazelcast.mc.disableHostnameVerification=true

3.7.3. Mutual Authentication

You can configure Management Center to require mutual authentication. With this setup, any client (be it a Web browser or an HTTP client such as curl) needs to present their TLS certificate and the Management Center needs to have its truststore configured so that the Management Center can know which clients it can trust. To enable mutual authentication, you need to use the following command line parameters when starting the Management Center:

-Dhazelcast.mc.tls.mutualAuthentication=REQUIRED

See the below snippet to see the full command to start the Management Center:

java -Dhazelcast.mc.tls.enabled=true \
     -Dhazelcast.mc.tls.trustStore=path to your truststore \
     -Dhazelcast.mc.tls.trustStorePassword=password for your truststore \
     -Dhazelcast.mc.tls.mutualAuthentication=REQUIRED \
     -jar hazelcast-management-center-4.0.war

The parameter -Dhazelcast.mc.tls.mutualAuthentication has two options:

  • REQUIRED: If the client does not provide a keystore or the provided keys are not included in the Management Center’s truststore, the client will not be authenticated.

  • OPTIONAL: If the client does not provide a keystore, it will be authenticated. But if the client provides keys that are not included in the Management Center’s truststore, the client will not be authenticated.

Managing Mutual Authentication Enabled Clusters

If mutual authentication is enabled for the cluster (as described here), the Management Center needs to have a keystore to identify itself. For this, you need to start the Management Center with the following command line parameters:

  • -Dhazelcast.mc.httpClient.tls.keyStore: Path to the keystore.

  • -Dhazelcast.mc.httpClient.tls.keyStorePassword: Password of the keystore.

  • -Dhazelcast.mc.httpClient.tls.keyStoreType: Type of the keystore. Its default value is JKS.

  • -Dhazelcast.mc.httpClient.tls.keyManagerAlgorithm: Name of the algorithm based on which the authentication keys are provided. System default is used if none is provided. You can find out the default by calling the javax.net.ssl.KeyManagerFactory#getDefaultAlgorithm method.

Excluding Specific TLS/SSL Protocols

When you enable TLS on the Management Center, it will support the clients connecting with any of the TLS/SSL protocols that the JVM supports by default. In order to disable specific protocols, you need to set the -Dhazelcast.mc.tls.excludeProtocols command line argument to a comma separated list of protocols to be excluded from the list of supported protocols. For example, to allow only TLSv1.2, you need to add the following command line argument when starting the Management Center:

-Dhazelcast.mc.tls.excludeProtocols=SSLv3,SSLv2Hello,TLSv1,TLSv1.1

When you specify the above argument, you should see a line similar to the following in the Management Center log:

2017-06-21 12:35:54.856:INFO:oejus.SslContextFactory:Enabled Protocols
[TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]

3.7.4. Using a Dictionary to Prevent Weak Passwords

In order to prevent certain words from being included in the user passwords, you can start the Management Center with -Dhazelcast.mc.security.dictionary.path command line parameter which points to a text file that contains a word on each line. As a result, the user passwords will not contain any dictionary words, making them harder to guess.

The words in the dictionary need to be at least three characters long in order to be used for checking the passwords. The shorter words are ignored to prevent them from blocking the usage of many password combinations. You can configure the minimum length of words by starting the Management Center with -Dhazelcast.mc.security.dictionary.minWordLength command line parameter and setting it to a number.

An example to start the Management Center using the aforementioned parameters is shown below:

java -Dhazelcast.mc.security.dictionary.path=/usr/MCtext/pwd.txt \
     -Dhazelcast.mc.security.dictionary.minWordLength=3 \
     -jar hazelcast-management-center-4.0.war

3.8. Configuring Logging

The Management Center uses Logback for its logging. By default, it uses the following configuration:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <statusListener class="ch.qos.logback.core.status.NopStatusListener" />

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>
                %d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n
            </pattern>
        </encoder>
    </appender>

    <logger level="WARN" name="org.eclipse.jetty">
        <appender-ref ref="STDOUT"/>
    </logger>
    <logger level="WARN" name="org.springframework">
        <appender-ref ref="STDOUT"/>
    </logger>
    <logger level="WARN" name="org.hibernate.validator">
        <appender-ref ref="STDOUT"/>
    </logger>
    <logger level="WARN" name="org.flywaydb">
        <appender-ref ref="STDOUT"/>
    </logger>
    <logger level="WARN" name="com.zaxxer.hikari">
        <appender-ref ref="STDOUT"/>
    </logger>
    <logger level="ERROR" name="org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver">
        <appender-ref ref="STDOUT"/>
    </logger>

    <root level="INFO">
        <appender-ref ref="STDOUT"/>
    </root>
</configuration>

To change the logging configuration, you can create a custom Logback configuration file and start the Management Center with the -Dlogback.configurationFile option pointing to your configuration file.

For example, you can create a file named logback-custom.xml with the following content and set logging level to DEBUG. To use this file as the logging configuration, you need to start the Management Center with the -Dlogback.configurationFile=/path/to/your/logback-custom.xml command line parameter:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>
                %d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n
            </pattern>
        </encoder>
    </appender>

    <root level="DEBUG">
        <appender-ref ref="STDOUT"/>
    </root>
</configuration>

To write log messages into rolling log files (in parallel with printing them into the console), you can use a similar Logback configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property name="pattern" value="%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n" />

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>${pattern}</pattern>
        </encoder>
    </appender>

    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${user.home}/mc-logs/mc.log</file>
        <!-- daily rollover with last 7 days history -->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${user.home}/mc-logs/mc.%d{yyyy-MM-dd}.log</fileNamePattern>
            <maxHistory>7</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>${pattern}</pattern>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="FILE"/>
    </root>
</configuration>

3.8.1. Enabling Audit Logging

You may enable additional security audit logging by setting the hazelcast.mc.auditlog.enabled system property to true. Log entries from the audit logging will be marked with the hazelcast.auditlog logging category.

An example log entry looks like the following:

2019-11-05 12:16:48 [qtp1551870003-37] INFO  hazelcast.auditlog - MC-2001 [Auth]:User logged in:{username=JohnHallaign}

MC-2001 [Auth] you see in this example represents the log’s type. The following table lists the current log categories along with their types:

Event Category Log Type/Description

Management Center Configuration Logs

  • MC-0001 [Config]: Time travel is enabled.

  • MC-0002 [Config]: Time travel is disabled.

  • MC-0003 [Config]: User is created.

  • MC-0004 [Config]: User is edited.

  • MC-0005 [Config]: User’s password is changed.

  • MC-0006 [Config]: User is deleted.

  • MC-0009 [Config]: License is set.

Cluster Configuration Logs

  • MC-1001 [Cluster Config]: Map’s configuration is changed.

  • MC-1003 [Cluster Config]: Cluster’s state is changed.

  • MC-1004 [Cluster Config]: Cluster is shut down.

  • MC-1005 [Cluster Config]: Member is shut down.

  • MC-1006 [Cluster Config]: Lite member is promoted.

  • MC-1007 [Cluster Config]: Cluster version is changed.

Authentication Logs

  • MC-2001 [Auth]: User logs in.

  • MC-2002 [Auth]: User logs out.

  • MC-2003 [Auth]: Login failures.

Scripting Logs

  • MC-3001 [Script]: Script is executed on a member.

Console Logs

  • MC-4001 [Console]: Console command is executed on the cluster.

Map/Cache Logs

  • MC-5001 [Browser]: User browses through a map screen in Management Center.

  • MC-5002 [Browser]: User browses through a cache screen in Management Center.

Hot Restart Logs

  • MC-6001 [Hot Restart]: Force start is run.

  • MC-6002 [Hot Restart]: Partial start is run.

  • MC-6003 [Hot Restart]: Hot Restart backup operation is triggered.

  • MC-6004 [Hot Restart]: Hot Restart backup operation is interrupted.

WAN Replication Logs

  • MC-7001 [WAN]: WAN configuration is added.

  • MC-7002 [WAN]: WAN consistency check operation is run.

  • MC-7003 [WAN]: WAN synchronization on a map is run.

  • MC-7004 [WAN]: State of the WAN publisher is changed.

  • MC-7005 [WAN]: Clear operation for the WAN events queue is run.

CP Subsystem Logs

  • MC-8001 [CP Subsystem]: Member is promoted to be a CP subsystem member.

  • MC-8002 [CP Subsystem]: Member is removed from CP subsystem.

  • MC-8003 [CP Subsystem]: CP subsystem is reset.

To write security audit logging into separate rolling log files, you can use a similar Logback configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property name="pattern" value="%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n" />

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>${pattern}</pattern>
        </encoder>
    </appender>

    <appender name="AUDIT_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${user.home}/mc-logs/audit.log</file>
        <!-- daily rollover with last 30 days history -->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${user.home}/mc-logs/audit.%d{yyyy-MM-dd}.log</fileNamePattern>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder>
            <pattern>${pattern}</pattern>
        </encoder>
    </appender>

    <logger level="INFO" name="hazelcast.auditlog">
        <appender-ref ref="AUDIT_FILE"/>
    </logger>

    <root level="INFO">
        <appender-ref ref="STDOUT"/>
    </root>
</configuration>

3.9. Using Variable Replacers

Variable replacers are used to replace custom strings during loading the configuration, either passed as command line arguments, used during UI based Management Center configuration, or configured with Management Center Configuration Tool. They can be used to mask sensitive information such as usernames and passwords. Of course their usage is not limited to security related information.

Variable replacers implement the interface com.hazelcast.webmonitor.configreplacer.spi.ConfigReplacer and they are configured via the following command line arguments:

  • -Dhazelcast.mc.configReplacer.class: Full class name of the replacer.

  • -Dhazelcast.mc.configReplacer.failIfValueMissing: Specifies whether the loading configuration process stops when a replacement value is missing. It is an optional attribute and its default value is true.

  • Additional command line arguments specific to each replacer implementation. All of the properties for the built-in replacers are explained in the upcoming sections.

The following replacer classes are provided by Hazelcast as example implementations of the ConfigReplacer interface. Note that you can also implement your own replacers.

  • EncryptionReplacer

  • PropertyReplacer

Each example replacer is explained in the following sections.

3.9.1. EncryptionReplacer

This example EncryptionReplacer replaces the encrypted variables with its plain form. The secret key for encryption/decryption is generated from a password which can be a value in a file and/or environment specific values, such as MAC address and actual user data.

Its full class name is com.hazelcast.webmonitor.configreplacer.EncryptionReplacer and the replacer prefix is ENC. Here are the properties used to configure this example replacer:

  • hazelcast.mc.configReplacer.prop.cipherAlgorithm: Cipher algorithm used for the encryption/decryption. Its default value is AES.

  • hazelcast.mc.configReplacer.prop.keyLengthBits: Length (in bits) of the secret key to be generated. Its default value is 128.

  • hazelcast.mc.configReplacer.prop.passwordFile: Path to a file whose content should be used as a part of the encryption password. When the property is not provided, no file is used as a part of the password. Its default value is null.

  • hazelcast.mc.configReplacer.prop.passwordNetworkInterface: Name of the network interface whose MAC address should be used as a part of the encryption password. When the property is not provided no network interface property is used as a part of the password. Its default value is null.

  • hazelcast.mc.configReplacer.prop.passwordUserProperties: Specifies whether the current user properties (user.name and user.home) should be used as a part of the encryption password. Its default value is true.

  • hazelcast.mc.configReplacer.prop.saltLengthBytes: Length (in bytes) of a random password salt. Its default value is 8.

  • hazelcast.mc.configReplacer.prop.secretKeyAlgorithm: Name of the secret key algorithm to be associated with the generated secret key. Its default value is AES.

  • hazelcast.mc.configReplacer.prop.secretKeyFactoryAlgorithm: Algorithm used to generate a secret key from a password. Its default value is PBKDF2WithHmacSHA256.

  • hazelcast.mc.configReplacer.prop.securityProvider: Name of a Java Security Provider to be used for retrieving the configured secret key factory and the cipher. Its default value is null.

Older Java versions may not support all the algorithms used as defaults. Use the property values supported by your Java version.

As a usage example, let’s create a password file and generate the encrypted strings out of this file as shown below:

  1. Create the password file: echo '/Za-uG3dDfpd,5.-' > /opt/master-password

  2. Define the encrypted variables:

    java -cp hazelcast-management-center-4.0.war \
         -Dhazelcast.mc.configReplacer.prop.passwordFile=/opt/master-password \
         -Dhazelcast.mc.configReplacer.prop.passwordUserProperties=false \
         com.hazelcast.webmonitor.configreplacer.EncryptionReplacer \
         "aPasswordToEncrypt"
    
    Output:
    
    $ENC{wJxe1vfHTgg=:531:WkAEdSi//YWEbwvVNoU9mUyZ0DE49acJeaJmGalHHfA=}
  3. Configure the replacer and provide the encrypted variables as command line arguments while starting the Management Center:

java \
 -Dhazelcast.mc.configReplacer.class=com.hazelcast.webmonitor.configreplacer.EncryptionReplacer \
 -Dhazelcast.mc.configReplacer.prop.passwordFile=/opt/master-password \
 -Dhazelcast.mc.configReplacer.prop.passwordUserProperties=false \
 -Dhazelcast.mc.tls.enabled=true \
 -Dhazelcast.mc.tls.keyStore=/opt/mc.keystore \
 -Dhazelcast.mc.tls.keyStorePassword='$ENC{wJxe1vfHTgg=:531:WkAEdSi//YWEbwvVNoU9mUyZ0DE49acJeaJmGalHHfA=}' \
 -jar hazelcast-management-center-4.0.war

3.9.2. PropertyReplacer

PropertyReplacer replaces variables by properties with the given name. Usually the system properties are used, e.g., ${user.name}.

Its full class name is com.hazelcast.webmonitor.configreplacer.PropertyReplacer and the replacer prefix is empty string ("").

3.9.3. Implementing Custom Replacers

You can also provide your own replacer implementations. All replacers have to implement the three methods that have the same signatures as the methods of the following interface:

import java.util.Properties;

public interface ConfigReplacer {
    void init(Properties properties);
    String getPrefix();
    String getReplacement(String maskedValue);
}

4. Connecting to IMDG Members

After you start and/or configure the Management Center service as explained in the Starting the Management Center Service and Configuring the Management Center chapters, make sure that http://localhost:8080 is up.

To connect the Management Center to your IMDG cluster, you have two options to be performed in Management Center:

  1. You can provide the IP addresses or hostnames of one or more IMDG members.

  2. Or, you can upload a configuration file in XML or YAML format. Management Center starts an IMDG client for each IMDG cluster. This configuration file is for that Management Center client. See the Launching Management Center chapter. You can think of the Management Center as a client connecting to the Hazelcast IMDG cluster.

If you enabled TLS/SSL on a Hazelcast IMDG cluster, then you need to perform the 2. step listed above, i.e., upload a configuration file. This configuration file needs to contain the parameters same as the ones you provide in the case when a client connects to a TLS/SSL enabled Hazelcast IMDG cluster.

5. Launching the Management Center User Interface

If you have the open source edition of Hazelcast IMDG, the Management Center can be used for at most three members in the cluster. To use it for more members, you need to have either a Management Center license, Hazelcast IMDG Enterprise license or Hazelcast IMDG Enterprise HD license. The license should be entered within the Management Center as described in the following paragraphs.

Even if you have a Hazelcast IMDG Enterprise or Enterprise HD license key and you set it as explained in the Setting the License Key section, you still need to enter this same license within the Management Center. See the following paragraphs to learn how you can enter your license.

Once you browse to http://localhost:8080 and since you are going to use the Management Center for the first time, the following dialog box appears:

Signing Up
If you already configured security before, a login dialog box appears instead.

It asks you to choose your security provider and create a username and password. Available security providers are Default, Active Directory, LDAP and JAAS, which are described in the following subsections.

Once you press the Save button, your security provider configuration is saved and you can log in with your credentials.

After you successfully login, the following page appears:

Starting Page

Now, you can create an IMDG cluster configuration. See the Managing Clusters chapter.

The Management Center can be used without a license if the cluster that you want to monitor has at most three members.

If you have a Management Center license or Hazelcast IMDG Enterprise license, you can enter it by clicking on the Administration button on the left menu and opening the Manage License tab. Here you can enter your license key and press the Update License button, as shown below:

Providing License for Management Center

Note that a license can likewise be provided using the system property hazelcast.mc.license (see the Starting with a License for details).

When you try to connect to a cluster that has more than three members without entering a license key or if your license key is expired, the following warning message is shown at the top:

Management Center License Warning

If you choose to continue without a license, please remember that the Management Center works if your cluster has at most three members.

The Management Center creates a folder with the name hazelcast-mc under your user’s home folder to save data files and above settings/license information. You can change the data folder by setting the hazelcast.mc.home system property.

5.1. Authentication Options

As mentioned above, available security providers are Default, Active Directory, LDAP and JAAS. They are described in the following below sections.

5.1.1. Default Authentication

You can use the default security provider for authentication/authorization on the Management Center. In this case, the user accounts are stored in the Management Center’s database.

Default Security Provider Configuration

Provide the details in this form for the default security provider:

  • Username: Username for the initial administrator user account.

  • Password, Confirm Password: Password for the initial administrator user account.

You can also use the create-user command in the MC Conf tool to configure the default security provider without any UI interactions. See this command’s description for details.

5.1.2. Active Directory Authentication

You can use your existing Active Directory server for authentication/authorization on the Management Center. In the "Configure Security" page, select Active Directory from the "Security Provider" combo box, and the following form page appears:

Active Directory Configuration

Provide the details in this form for your Active Directory server:

  • URL: URL of your Active Directory server, including the schema (ldap:// or ldaps://) and port.

  • Domain: Domain of your organization on Active Directory.

  • User Search Filter: LDAP search filter expression to search for the users. {0} will be replaced with username@domain and {1} will be replaced with only the username. You can use both placeholders, only one of them or none in your search filter. For example, (&(objectClass=user)(userPrincipalName={0})) searches for a username that matches with the userPrincipalName attribute and member of the object class user.

  • Admin Group(s): Members of this group and its nested groups have admin privileges on the Management Center. To use more than one group, separate them with a semicolon (;).

  • User Group(s): Members of this group and its nested groups have read and write privileges on the Management Center. To use more than one group, separate them with a semicolon (;).

  • Read-only User Group(s): Members of this group and its nested groups have only read privilege on the Management Center. To use more than one group, separate them with a semicolon (;).

  • Metrics-only Group(s): Members of this group and its nested groups have the privilege to see only the metrics on Management Center. To use more than one group, separate them with a semicolon (;).

  • Nested Group Search: Disable if you have a large LDAP group structure and it takes a long time to query all nested groups during login.

  • Test Username: Username to test the Active Directory configuration with. Note that this value will not be saved and only be used for testing the Active Directory configuration.

  • Test Password: Password to test the Active Directory configuration with. Note that this value will not be saved and only be used for testing the Active Directory configuration.

Before saving the configuration, you can test it by clicking the Test button. Note that the user you test with needs to be a member of one of the groups you have configured for the Management Center.

You can use the hazelcast.mc.ldap.timeout system property to specify both connect and read timeout values for Active Directory search queries. It is in milliseconds and its default value is 3000 milliseconds.
Configuration Hot Reload

Once configured, Active Directory settings are saved in a local database managed by Management Center. If you need to update your settings afterwards, you need to provide the import properties file under <hazelcast-mc>/import/securityHotReload.properties, and then click on the Reload Security Config button on the login page. The securityHotReload.properties should contain the following properties:

url=<active directory instance url>
domain=<domain>
adminGroup=<Admin group(s). Use ';' to separate multiple groups>
userGroup=<Read-write group(s). Use ';' to separate multiple groups>
readonlyUserGroup<Read-only group(s). Use ';' to separate multiple groups>
metricsOnlyGroup<Metrics-only group(s). Use ';' to separate multiple groups>
The Reload Security Config button will only appear when the <hazelcast-mc>/import/securityHotReload.properties file is present. After a successful import, the file will be renamed as importedSecurityHotReload-<import_timestamp>.properties.bak.

Alternatively, you can use MC-Conf tool’s security reset and active-directory configure tasks to configure the Active Directory security provider from scratch, but you need to stop the Management Center service for this configuration option. See the Management Center Configuration Tool section for more information.

5.1.3. JAAS Authentication

You can use your own javax.security.auth.spi.LoginModule implementation for authentication/authorization on the Management Center. In the "Configure Security" page, select JAAS from the "Security Provider" combo box, and the following page appears:

JAAS Configuration

Provide the details in this form for your JAAS LoginModule implementation:

  • Login Module Class: Fully qualified class name of your javax.security.auth.spi.LoginModule implementation.

  • Admin Group: Members of this group have admin privileges on the Management Center.

  • User Group: Members of this group have read and write privileges on the Management Center.

  • Read-only User Group: Members of this group have only read privilege on the Management Center.

  • Metrics-only Group: Members of this group have the privilege to see only the metrics on the Management Center.

The following is an example implementation. Note that we return two java.security.Principal instances; one of them is the username and the other one is a group name, which you will use when configuring JAAS security as described above.

import javax.security.auth.Subject;
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.CallbackHandler;
import javax.security.auth.callback.NameCallback;
import javax.security.auth.callback.PasswordCallback;
import javax.security.auth.login.LoginException;
import javax.security.auth.spi.LoginModule;
import java.security.Principal;
import java.util.Map;

public class SampleLoginModule implements LoginModule {
    private Subject subject;
    private String password;
    private String username;

    @Override
    public void initialize(Subject subject, CallbackHandler callbackHandler, Map<String, ?> sharedState, Map<String, ?> options) {
        this.subject = subject;

        try {
            NameCallback nameCallback = new NameCallback("prompt");
            PasswordCallback passwordCallback = new PasswordCallback("prompt", false);

            callbackHandler.handle(new Callback[] {nameCallback, passwordCallback });

            password = new String(passwordCallback.getPassword());
            username = nameCallback.getName();
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

    @Override
    public boolean login() throws LoginException {
        if (!username.equals("emre")) {
            throw new LoginException("Bad User");
        }

        if (!password.equals("pass1234")) {
            throw new LoginException("Bad Password");
        }

        subject.getPrincipals().add(new Principal() {
            public String getName() {
                return "emre";
            }
        });

        subject.getPrincipals().add(new Principal() {
            public String getName() {
                return "HazelcastMCAdmin";
            }
        });

        return true;
    }

    @Override
    public boolean commit() throws LoginException {
        return true;
    }

    @Override
    public boolean abort() throws LoginException {
        return true;
    }

    @Override
    public boolean logout() throws LoginException {
        return true;
    }
}

5.1.4. LDAP Authentication

You can use your existing LDAP server for authentication/authorization on the Management Center. In the "Configure Security" page, select LDAP from the "Security Provider" combo box, and the following form page appears:

LDAP Configuration

Provide the details in this form for your LDAP server:

  • URL: URL of your LDAP server, including schema (ldap:// or ldaps://) and port.

  • Distinguished name (DN) of user: DN of a user that has admin privileges on the LDAP server. It is used to connect to the server when authenticating users.

  • User DN: DN to be used for searching users.

  • Group DN: DN to be used for searching groups.

  • Admin Group(s): Members of this group and its nested groups have admin privileges on the Management Center. To use more than one group, separate them with a semicolon (;).

  • User Group(s): Members of this group and its nested groups have read and write privileges on the Management Center. To use more than one group, separate them with a semicolon (;).

  • Read-only User Group(s): Members of this group and its nested groups have only read privilege on the Management Center. To use more than one group, separate them with a semicolon (;).

  • Metrics-only Group(s): Members of this group and its nested groups have the privilege to see only the metrics on the Management Center. To use more than one group, separate them with a semicolon (;).

  • Start TLS: Enable if your LDAP server uses Start TLS operation.

  • User Search Filter: LDAP search filter expression to search for the users. For example, uid={0} searches for a username that matches with the uid attribute.

  • Group Search Filter: LDAP search filter expression to search for the groups. For example, uniquemember={0} searches for a group that matches with the uniquemember attribute.

  • Nested Group Search: Disable if you have a large LDAP group structure and it takes a long time to query all nested groups during login.

Values for Admin, User, Read-only and Metrics-Only group names must be given as plain names. They should not contain any LDAP attributes such as CN, OU and DC.
You can use the hazelcast.mc.ldap.timeout system property to specify connect and read timeout values for LDAP search queries. It is in milliseconds and its default value is 3000 milliseconds.
Configuration Hot Reload

Once configured, LDAP settings are saved in a local database managed by Management Center. If you need to update your settings afterwards, you need to provide the import properties file under <hazelcast-mc>/import/securityHotReload.properties, and then click on the Reload Security Config button on the login page. The securityHotReload.properties should contain the following properties:

url=<Ldap URL>
username=<Distinguished name (DN) of user>
password=<password>
userDn=<User DN>
groupDn<Group DN>
startTls=<true|false>
adminGroup=<Admin group(s). Use ';' to separate multiple groups>
userGroup=<Read-write group(s). Use ';' to separate multiple groups>
readonlyUserGroup<Read-only group(s). Use ';' to separate multiple groups>
metricsOnlyGroup<Metrics-only group(s). Use ';' to separate multiple groups>
userSearchFilter=<User Search Filter>
groupSearchFilter=<Group Search Filter>
nestedGroupSearch=<true|false>
The Reload Security Config button will only appear when the <hazelcast-mc>/import/securityHotReload.properties file is present. After a successful import, the file will be renamed as importedSecurityHotReload-<import_timestamp>.properties.bak.

Alternatively, you can use MC-Conf tool’s security reset and active-directory configure tasks to configure the Active Directory security provider from scratch, but you need to stop the Management Center service for this configuration option. See the Management Center Configuration Tool section for more information.

Enabling TLS/SSL for LDAP

If your LDAP server is using ldaps (LDAP over SSL) protocol or the Start TLS operation, use the following command line parameters for your Management Center deployment:

  • -Dhazelcast.mc.ldap.ssl.trustStore: Path to the truststore. This truststore needs to contain the public key of your LDAP server.

  • -Dhazelcast.mc.ldap.ssl.trustStorePassword: Password of the truststore.

  • -Dhazelcast.mc.ldap.ssl.trustStoreType: Type of the truststore. Its default value is JKS.

  • -Dhazelcast.mc.ldap.ssl.trustManagerAlgorithm: Name of the algorithm based on which the authentication keys are provided. System default is used if none is provided. You can find out the default by calling the javax.net.ssl.TrustManagerFactory#getDefaultAlgorithm method.

Password Encryption

By default, the password that you use in the LDAP configuration is stored in a plain text in a local database. This might pose a security risk. To store the LDAP password in an encrypted form, we offer the following options:

  • Provide a keystore password: This creates and manages a Java keystore under the Management Center home directory. The LDAP password is stored in this keystore in an encrypted form.

  • Configure an external Java keystore: This uses an existing Java keystore. This option might also be used to store the password in an HSM that provides a Java keystore API.

In the case of using either one of the options, the LDAP password you enter on the initial configuration UI dialog will be stored in an encrypted form in a Java keystore instead of the local database.

You can also encrypt the password, and provide it in an encrypted form, when configuring LDAP security provider. See the Variable Replacers section for more information.
Providing a Master Key for Encryption

There are two ways to provide a master key for encryption:

  • If you deploy the Management Center on an application server, you need to set the MC_KEYSTORE_PASS environment variable before starting Management Center. This option is less secure. You should clear the environment variable once you make sure you can log in with your LDAP credentials to minimize the security risk.

  • If you’re starting the Management Center from the command line, you can start it with -Dhazelcast.mc.askKeyStorePassword. The Management Center asks for the keystore password upon start and use it as a password for the keystore it creates. This option is more secure as it only stores the keystore password in the memory.

By default, the Management Center creates a Java keystore file under the Management Center home directory with the name mc.jceks. You can change the location of this file by using the -Dhazelcast.mc.keyStore.path=/path/to/keyStore.jceks JVM argument.

Configuring an External Java KeyStore

If you don’t want the Management Center to create a keystore for you and use an existing one that you’ve created before (or an HSM), set the following JVM arguments when starting the Management Center:

  • -Dhazelcast.mc.useExistingKeyStore=true: Enables use of an existing keystore.

  • -Dhazelcast.mc.existingKeyStore.path=/path/to/existing/keyStore.jceks: Path to the keystore. You do not have to set it if you use an HSM.

  • -Dhazelcast.mc.existingKeyStore.pass=somepass: Password for the keystore. You do not have to set it if HSM provides another means to unlock HSM.

  • -Dhazelcast.mc.existingKeyStore.type=JCEKS: Type of the keystore.

  • -Dhazelcast.mc.existingKeyStore.provider=com.yourprovider.MyProvider: Provider of the keystore. Leave empty to use the system provider. Specify the class name of your HSM’s java.security.Provider implementation if you use an HSM.

Make sure your keystore supports storing `SecretKey`s.
Updating Encrypted Passwords

You can use the update-ldap-password command in the MC Conf tool to update the encrypted LDAP password stored in the keystore. See this command’s description for details.

6. User Interface Overview

Once the page is loaded after selecting a cluster, Dashboard Page appears as shown below:

Dashboard Page

This page provides the fundamental properties of the selected cluster which are explained in the Dashboard Page chapter.

The user interface has a toolbar on the top and a menu on the left.

6.1. Toolbar

Management Center Toolbar

The toolbar has the following elements:

  • Docs: Opens the Management Center documentation in a new browser tab.

  • Time Travel: Shows the cluster’s situation at a time in the past. See the Time Travel section.

  • Console: Opens the in-page console so that you can easily execute commands in your cluster. See the Executing Console Commands chapter. Note that this top menu item appears only when you select a cluster to monitor.

  • User name: Shows the current user’s name. When you hover your mouse cursor onto it, the user’s last login time is shown as a tooltip, for security purposes.

  • Cluster Selector: Switches between the clusters. When clicked on, a dropdown list of available clusters appears. You can select any cluster; once selected, the page immediately loads with the selected cluster’s information. This selector has the same functionality as selecting a cluster in the "Manage Clusters" menu of the user interface.

  • Logout: Closes the current user’s session.

The Management Center user interface includes a menu on the left which lists the data structures in the cluster, cluster members and clients connected to the cluster (numbers in square brackets show the instance count for each entity), as shown below. You can also see an overview state of your cluster, execute commands and scripts, and perform user/license operations using this menu:

Management Center Left Menu
WAN Replication button is only visible with the Hazelcast IMDG Enterprise license.

The menu consists of the following parent menu headings:

  • Management Center: Includes menu items to manage clusters, user and license operations, and examine the user interface logs.

  • Cluster: Includes menu items for general cluster operations, monitoring clients/cluster members and executing scripts/commands.

  • Storage: Includes menu items to monitor data structures such as the maps, caches and PN counters in your cluster.

  • Messaging: Includes menu items to monitor queues and topics in your cluster.

  • Compute: Includes the "Executors" menu item to monitor the executor services in your cluster.

If you have not selected a cluster to monitor, only the "Management Center" menu items are shown in the user interface.

The following is the list of all menu items with links to their explanations:

7. Managing Clusters

"Manage Clusters" is the first page that appears after you login to Management Center. It is used to add, remove or edit cluster configurations, and looks like the following:

Starting Page

If you are using Management Center for the first time, you need to add your cluster’s configuration using this page You can always go to this page during your Management Center usage by clicking on the "Manage Clusters" menu item under the "Management Center" parent menu heading.

To start monitoring your cluster you have already created (on your local machine, another development environment, etc.) you need to first define a cluster configuration to connect Management Center to your cluster. After you click on the "Add Cluster Config" button shown in the above screenshot, you can do this by either of the following options:

  • Using a form by providing the cluster name and member addresses

  • Uploading a configuration file

7.1. Creating a Cluster Configuration Using Form

Click on the "Add Cluster Config" button as seen in the above screenshot. The following page appears:

Add New Cluster Configuration
  • Cluster Name: Here, you enter the name of your cluster, which you defined while you were creating it. If you haven’t specified anything, dev is the default name for a cluster.

  • Cluster Config State: Select "Enabled" to apply the cluster configuration. You can also select "Disabled" just to create the cluster configuration and use it at a later time. In this case, the configuration is ignored until you enable it.

  • Member Addresses: Give the cluster members' addresses (one or preferably more) in the form of hostname, hostname:port, IP address or IP address:port. If you don’t provide a port, the system tries the ports through 5701 and 5703 in a random order. Type the member address and press enter (or return) to make it visible in the "Member Addresses" field.

Click on the Save button and the "Manage Clusters" page loads with the new cluster configuration added, as shown below:

Cluster Configuration Added

Here, you can see the state of the cluster and member count. Also, at the top left of the cluster box, you see the green colored dot, meaning that Management Center is connected to your cluster.

You can add multiple cluster configurations using the same mechanism as explained above.

The following page shows an example where Management Center is and is not able to connect one of clusters:

Manage Clusters

You can edit or delete a cluster configuration using the icons on any cluster’s box area in this page.

To see the status of a cluster, click on its Select button. The "Dashboard" page of the related cluster will load; see the Dashboard Page chapter for the details. Clicking on the Select button and selecting a cluster in the cluster selector (located at the toolbar) have the same functionality.

7.2. Creating a Cluster Configuration by Uploading File

Alternatively, you can upload a configuration file for the Management Center client created for your IMDG cluster. For this, click on the Upload Config File button which is seen after you click on the "Add Cluster Config" button. The following page appears:

Upload Config File

The name of your cluster will be automatically generated as specified in the configuration file you upload. As mentioned before in the connecting members chapter, you can think of Management Center as a client that connects to your IMDG cluster you create. Therefore, this configuration file is a well-known hazelcast.client in XML or YAML language.

Here is an example configuration file:

Example hazelcast-client.xml file
<hazelcast-client>
    <cluster-name>dev</cluster-name>
    <network>
        <cluster-members>
            <address>127.0.0.1</address>
            <address>127.0.0.2</address>
        </cluster-members>
    </network>
</hazelcast-client>

You can simply drag this file onto the Cluster Config File box in this page, or find it using the browse link in the same box.

Select "Enabled" in the Cluster Config State combobox to apply the configuration. You can also select "Disabled" just to create the configuration and use it at a later time. In this case, the configuration is ignored until you enable it.

You can always go back to the form page (explained in the previous section) by clicking on the Add via Form on this page.

8. Managing Users

User management is only available for the default security provider. See the Default Authentication section for more information.
Users

To add a user to the system, specify the username, e-mail and password in the Add/Edit User part of the page. Then provide the user’s privilege status using the checkboxes under Permissions:

  • Admin: Select if the user to be added will have the administrator privileges.

  • Read/Write: If this permission is given to the user, Home, Scripting, Console, Documentation and Time Travel items will be visible. The users with this permission can update a map configuration and perform operations on the members.

  • Read: If this permission is given to the user, only Home, Documentation and Time Travel items will be visible at the toolbar on that user’s session. Also, the users with this permission cannot update a map configuration, run a garbage collection and take a thread dump on a cluster member, or shutdown a member (see the Monitoring Members section).

  • Metrics Only: If this permission is given to the user, only Home, Documentation and Time Travel items will be visible at the toolbar on that user’s session. Also, the users with this permission cannot browse a map or a cache to see their contents, cannot update a map configuration, run a garbage collection and take a thread dump on a cluster member, or shutdown a member (see the Monitoring Members section).

After you enter/select all the fields, click on the Save button to create the user. You will see the newly created user’s username on the right side, in the List of Users part of the page.

To edit or delete a user, click on its username listed in the List of Users. Selected user information appears on the left side of the page. To update the user information, change the fields as desired and click on the Save button; note that you can only change the username and/or its privilege.

You can also change a user’s password or delete the user account. To change the user’s password, click on the Change Password button while editing the user. To delete the user from the system, click on the Delete button. Note that changing the password of a user and deleting the user account both require you to enter your own password.

Certain user management operations are also available in the MC Conf tool. See the Management Center Configuration Tool section for more information.

9. Managing Licenses

You can manage the following types of licenses in Management Center:

  • License for Management Center

  • License for your Hazelcast product

9.1. Management Center License

Using the "License" menu item under the "Management Center" parent menu heading, you can view the details of your Management Center license. An example screenshot is shown below.

License Screen

It shows the expiration date, total licensed member count and type of your Management Center license.

For security reasons, the license key itself is not shown. Instead, SHA-256 hash of the key as a Base64 encoded string is shown.

If there are any problems related to your Management Center license, "License" menu item is highlighted with red exclamation points. In this case, please check this screen to see what the problem is. The following are the possible problems:

  • Your Management Center license is expired.

  • The count of your cluster members is more than the allowed count by the license.

If the Management Center license expires in 30 days or less, or has already expired, a warning will appear in the upper right corner once a day. Warning will contain time remaining until license expires or how long the license is expired. There will also be two buttons - the first one with label "Show License" will redirect you to the license page, the second one with label "Dismiss Alert" will dismiss the alert. An example screenshot is shown below.

License Expiration Alert

To update the Management Center license, you can open the Manage License tab and click on the Update License button and enter the new license code.

License

Alternatively, a license can be provided using the hazelcast.mc.license system property (see the Starting with a License section for details).

9.2. Cluster License

Using the "License" menu item under the "Cluster" parent menu heading, you can view the details of your cluster’s license (Hazelcast IMDG license). An example screenshot is shown below.

License Screen

It shows the expiration date, total licensed member count and type of your Hazelcast IMDG license.

For security reasons, the license key itself is not shown. Instead, SHA-256 hash of the key as a Base64 encoded string is shown.

If there are any problems related to your cluster license, "License" menu item is highlighted with red exclamation points. In this case, please check this screen to see what the problem is. The following are the possible problems:

  • Your Hazelcast IMDG license is expired.

  • The count of your cluster members is more than the allowed count by the license.

If the cluster license expires in 30 days or less, or has already expired, a warning will appear in the upper right corner once a day, similar to the one for the Management Center license. Warning will contain time remaining until license expires or how long the license is expired. There will also be two buttons - the first one with label "Show License" will redirect you to the license page, the second one with label "Dismiss Alert" will dismiss the alert.

10. Dashboard Page

This is the first page appearing after logging in. It gives an overview of the connected cluster. The following subsections describe each portion of the page.

10.1. Memory Utilization

This part of the page provides information related to the memory usages for each member, as shown below:

Memory Utilization

The first column lists the members with their IPs and ports. The next columns show the used and free memories out of the total memory reserved for Hazelcast IMDG usage, in real-time. The Max. Heap column lists the maximum memory capacity of each member and the Heap Usage Percentage column lists the percentage value of used memory out of the maximum memory. The Used Heap column shows the memory usage of members graphically. When you move the mouse cursor on a desired graph, you can see the memory usage at the time where the cursor is placed. Graphs under this column show the memory usages approximately for the last 2 minutes.

10.2. Heap Memory Distribution

This part of the page graphically provides the cluster wise breakdown of heap memory, as shown below. The blue area is the heap memory used by the maps (including all owned/backup entries, any near cache usage and cost of the Merkle tree). The dark yellow area is the heap memory used by both non-Hazelcast entities and all Hazelcast entities except the map, i.e., the heap memory used by all entities subtracted by the heap memory used by map. The green area is the free heap memory out of the whole cluster’s total committed heap memory.

Heap Memory Distribution of Cluster

In the above example, you can see about 3% of the total heap memory is used by Hazelcast IMDG maps, about 30% is used by both non-Hazelcast entities and all Hazelcast entities except the map and the rest of the total heap memory is free. You can see the exact percentages by placing the mouse cursor on the chart.

10.3. Cluster State/Health/Client Filtering/CP Subsystem

This part has the following status indicator elements:

  • Cluster State: Shows the current cluster state. For more information on cluster states, see the Cluster State section.

  • Cluster Health: Shows how many migrations are taking place currently.

  • Cluster Filtering: Shows values for the current cluster client filtering status and type. For more information on the cluster client filtering, see the Changing Cluster Client Filtering section.

  • CP Subsystem: Shows the CP subsystem status. For more information on the CP subsystem support in the Management Center, see the CP Subsystem section.

Cluster client filtering is only available with Hazelcast IMDG license that includes the Cluster Client Filtering feature.
Cluster State and Cluster Health

10.4. Partition Distribution

This chart shows what percentage of partitions each cluster member has, as shown below:

Partition Distribution per Member

You can see each member’s partition percentages by placing the mouse cursor on the chart. In the above example, you can see that each member has about 33.3% of the total partition count (which is 271 by default and configurable; see the hazelcast.partition.count property explained in the System Properties section in the Hazelcast IMDG Reference Manual).

The partition distribution chart does not show any information until you create your distributed objects. When you add new members to your cluster, there will be no partition migration since partitions do not exist yet. Once you connect to your cluster and, for example, create a map (using hazelcastInstance.getMap()), only then this chart starts to show partition distribution information.

10.5. CPU Utilization

This part of the page provides load and utilization information for the CPUs for each cluster member, as shown below:

CPU Utilization

The first column lists the members with their IPs and ports. The next columns list the system load averages on each member for the last 1, 5 and 15 minutes. These average values are calculated as the sum of the count of runnable entities running on and queued to the available CPUs averaged over the last 1, 5 and 15 minutes. This calculation is operating system specific, typically a damped time-dependent average. If system load average is not available, these columns show negative values.

The last column (Utilization(%)) graphically shows the recent load on the CPUs. When you move the mouse cursor on a chart, you can see the CPU load at the time where the cursor is placed. The charts under this column shows the CPU loads approximately for the last 2 minutes. If recent CPU load is not available, you will see N/A values.

11. Monitoring Members

Use this menu item to monitor each cluster member and perform operations like running garbage collection (GC) and taking a thread dump.

You can see a list of all the members in your cluster by clicking on the Members menu item on the left panel. A new page is opened on the right, as shown below.

Member Grid View
You may see a warning icon with exclamation mark in the list when your runtime or hardware configuration does not follow the performance recommendations. See IMDG Deployment and Operations Guide for more information.

You can filter the members shown and you can also sort the table by clicking on the column headers. Members that participate in the CP subsystem are marked with the CP icon. Clicking on a member name opens a new page for monitoring that member on the right, as shown below.

Monitoring Members

You can perform the following operations on the selected member using the buttons located at the top left of the page:

  • Run GC: Executes garbage collection on the selected member. A notification stating that the GC execution was successful is shown.

  • Thread Dump: Takes a thread dump of the selected member and shows it in a separate dialog.

  • Shutdown Member: Shuts down the selected member.

  • Promote Member: It is only shown for the lite members. When pressed, the lite member becomes a data member.

Shutdown member operation requires enabled REST API in the IMDG cluster. See the IMDG documentation for more information.

Next to the above operation buttons, you can see the informative buttons as described below:

  • Number of Owned Partitions: Shows how many partitions are assigned to the selected member.

  • Member Version: Shows the Hazelcast IMDG cluster version which the selected member belongs to.

  • CP Member UUID: Shows CP member UUID if the member participates in the CP subsystem of the cluster.

The CPU Utilization chart shows the percentage of CPU usage on the selected member. The Heap/Memory Utilization charts show the memory usage on the selected member with three different metrics (maximum, used and total memory). You can open each chart as a separate dialog using the maximize button placed at top right of them; this gives you a clearer view of the chart.

Runtime is a dynamically updated window tab showing the processor number, the start and up times, and the maximum, total and free memory sizes of the selected member. These values are collected from the default MXBeans provided by the Java Virtual Machine (JVM). Descriptions from the Javadocs and some explanations are below:

  • Number of Processors: Number of processors available to the member (JVM).

  • Start Time: Start time of the member (JVM) in milliseconds.

  • Up Time: Uptime of the member (JVM) in milliseconds

  • Maximum Memory: Maximum amount of memory that the member (JVM) will attempt to use.

  • Free Memory: Amount of free memory in the member (JVM).

  • Used Heap Memory: Amount of used memory in bytes.

  • Max Heap Memory: Maximum amount of memory in bytes that can be used for memory management.

  • Used Non-Heap Memory: Amount of used memory in bytes.

  • Max Non-Heap Memory: Maximum amount of memory in bytes that can be used for memory management.

  • Total Loaded Classes: Total number of classes that have been loaded since the member (JVM) has started execution.

  • Current Loaded Classes: Number of classes that are currently loaded in the member (JVM).

  • Total Unloaded Classes: Total number of classes unloaded since the member (JVM) has started execution.

  • Total Thread Count: Total number of threads created and also started since the member (JVM) started.

  • Active Thread Count: Current number of live threads including both daemon and non-daemon threads.

  • Peak Thread Count: Peak live thread count since the member (JVM) started or peak was reset.

  • Daemon Thread Count: Current number of live daemon threads.

  • OS: Free Physical Memory: Amount of free physical memory in bytes.

  • OS: Committed Virtual Memory: Amount of virtual memory that is guaranteed to be available to the running process in bytes.

  • OS: Total Physical Memory: Total amount of physical memory in bytes.

  • OS: Free Swap Space: Amount of free swap space in bytes. Swap space is used when the amount of physical memory (RAM) is full. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space.

  • OS: Total Swap Space: Total amount of swap space in bytes.

  • OS: Maximum File Descriptor Count: Maximum number of file descriptors. File descriptor is an integer number that uniquely represents an opened file in the operating system.

  • OS: Open File Descriptor Count: Number of open file descriptors.

  • OS: Process CPU Time: CPU time used by the process on which the member (JVM) is running in nanoseconds.

  • OS: Process CPU Load: Recent CPU usage for the member (JVM) process. This is a double with a value from 0.0 to 1.0. A value of 0.0 means that none of the CPUs were running threads from the member (JVM) process during the recent period of time observed, while a value of 1.0 means that all CPUs were actively running threads from the member (JVM) 100% of the time during the recent period being observed. Threads from the member (JVM) include the application threads as well as the member (JVM) internal threads.

  • OS: System Load Average: System load average for the last minute. The system load average is the average over a period of time of this sum: (the number of runnable entities queued to the available processors) + (the number of runnable entities running on the available processors). The way in which the load average is calculated is operating system specific but it is typically a damped time-dependent average.

  • OS: System CPU Load: Recent CPU usage for the whole system represented as a percentage value. 0% means that all CPUs were idle during the recent period of time observed, while 100% means that all CPUs were actively running during the recent period being observed.

These descriptions may vary according to the JVM version or vendor.

Next to the Runtime tab, the Properties tab shows the system properties.

The Member Configuration window shows the XML configuration of the connected Hazelcast cluster.

The List of Slow Operations gives an overview of detected slow operations which occurred on that member. The data is collected by the SlowOperationDetector.

List of Slow Operations

Click on an entry to open a dialog which shows the stacktrace and detailed information about each slow invocation of this operation.

Slow Operations Details

12. Monitoring Clients

You can use the Clients menu item to monitor all the clients that are connected to your Hazelcast cluster.

Only basic information for clients, like client instance name, address, type and labels, is shown by default. The values for other fields are shown as N/A. As a prerequisite for seeing the full information, you need to enable the client statistics before starting your clients. This can be done by setting the hazelcast.client.statistics.enabled system property to true on the client. Please see the Client System Properties section in the Hazelcast IMDG Reference Manual for more information. After you enable the client statistics, you can monitor your clients using Hazelcast Management Center.

You can see a list of all the clients in your cluster by clicking on the Clients menu item on the left panel. A new page is opened on the right, as shown below. The page has two tabs: Connection and Filter. The Connection tab is opened by default. This tab shows the list of all the clients. See the Changing Cluster Client Filtering section for the Filter tab’s description.

Client Grid View

By default, hostname of the client is shown in the address column. You can change it to show its IP address or its canonical hostname instead by using the Address Type combo box. Note that this will also cause the client details page to show the IP address or the canonical hostname.

You can filter the clients shown and you can also sort the table by clicking on the column headers. Clicking on a client name will open a new page for monitoring that client on the right, as shown below.

Monitoring Client Detailed

The Heap Memory Utilization chart shows the memory usage on the selected client with three different metrics (maximum, used and total memory) represented by different colors. You can open this chart as a separate window using the maximize button placed at top right of it; this gives you a clearer view of the chart.

General is a dynamically updated window tab showing general information about the client. Below are brief explanations for each piece of information:

  • Name: Name of the client instance.

  • Address: Local IP address of the client that is used for connecting to members.

  • Type: Type of the client.

  • Enterprise: Yes, if the client is an Hazelcast IMDG Enterprise client.

  • Member Connection: Shows to which member a client is currently connected to. Please note that ALL means a client is configured so that it might connect to all members of a cluster, i.e., it might not have a connection to all members all the time.

  • Version: Version of the client.

  • Last Connection to Cluster: Time that the client connected to the cluster. It is reset on each reconnection.

  • Last Statistics Collection: Time when the latest update for the statistics is collected from the client.

  • User Executor Queue Size: Number of waiting tasks in the client user executor.

  • Labels: List of client labels.

Next to the General tab, the Runtime tab shows the processor number, uptime, and maximum, total and free memory sizes of the selected client. These values are collected from the default MXBeans provided by the Java Virtual Machine (JVM). Descriptions from the Javadocs and some explanations are below:

  • Number of Processors: Number of processors available to the client (JVM).

  • Up Time: Uptime of the client (JVM).

  • Maximum Memory: Maximum amount of memory that the client (JVM) will attempt to use.

  • Total Memory: Amount of total heap memory currently available for current and future objects in the client (JVM).

  • Free Memory: Amount of free heap memory in the client (JVM).

  • Used Memory: Amount of used heap memory in the client (JVM).

Next to the Runtime tab, the OS tab shows statistics about the operating system of the client. These values are collected from the default MXBeans provided by the Java Virtual Machine (JVM). Descriptions from the Javadocs and some explanations are below:

  • Free Physical Memory: Amount of free physical memory.

  • Committed Virtual Memory: Amount of virtual memory that is guaranteed to be available to the running process.

  • Total Physical Memory: Total amount of physical memory.

  • Free Swap Space: Amount of free swap space. Swap space is used when the amount of physical memory (RAM) is full. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space.

  • Total Swap Space: Total amount of swap space.

  • Maximum File Descriptor Count: Maximum number of file descriptors. File descriptor is an integer number that uniquely represents an opened file in the operating system.

  • Open File Descriptor Count: Number of open file descriptors.

  • Process CPU Time: CPU time used by the process on which the member (JVM) is running.

  • System Load Average: System load average for the last minute. The system load average is the average over a period of time of this sum: (the number of runnable entities queued to the available processors) + (the number of runnable entities running on the available processors). The way in which the load average is calculated is operating system specific but it is typically a damped time-dependent average.

Some of the Runtime/OS statistics may not be available for your client’s JVM implementation/operating system. N/A is shown for these types of statistics. Please refer to your JVM/operating system documentation for further details.

The Client Near Cache Statistics table shows statistics related to the Near Cache of a client. There are two separate tables; one for maps and one for caches.

  • Map/Cache Name: Name of the map or cache.

  • Creation Time: Creation time of this Near Cache on the client.

  • Evictions: Number of evictions of Near Cache entries owned by the client.

  • Expirations: Number of TTL and max-idle expirations of Near Cache entries owned by the client.

  • Hits: Number of hits (reads) of Near Cache entries owned by the client.

  • Misses: Number of misses of Near Cache entries owned by the client.

  • Owned Entry Count: Number of Near Cache entries owned by the client.

  • Owned Entry Memory Cost: Memory cost of Near Cache entries owned by the client.

  • LP Duration: Duration of the last Near Cache key persistence (when the pre-load feature is enabled).

  • LP Key Count: Number of Near Cache key persistences (when the pre-load feature is enabled).

  • LP Time: Time of the last Near Cache key persistence (when the pre-load feature is enabled).

  • LP Written Bytes: Written number of bytes of the last Near Cache key persistence (when the pre-load feature is enabled).

  • LP Failure: Failure reason of the last Near Cache persistence (when the pre-load feature is enabled).

Please note that you can configure the time interval for which the client statistics are collected and sent to the cluster, using the system property hazelcast.client.statistics.period.seconds. See the System Properties section in the Hazelcast IMDG Reference Manual for more information.

12.1. Changing Cluster Client Filtering

The Filter tab is only available with Hazelcast IMDG license that includes the Cluster Client Filtering feature.

The Filter tab includes Cluster Client Filtering status, Cluster Client Filter Settings and Client Filtering Lists sections, as shown below.

Client Filtering View

The Cluster Client Filtering status section describes if there is a deployed client filtering list available to all cluster members (Enabled status), or if the feature is disabled for the cluster and the members allow any clients (Disabled status).

The Cluster Client Filter Settings section allows to specify the status of the feature and the filtering type and to deploy any modifications made in client filtering lists to the deployed list available to all cluster members. On the deploy action the following happens:

  • If the status to be deployed is Disabled, the deployed client filtering list available to all cluster members is cleaned up and the members start allowing any client to connect.

  • If the status to be deployed is Enabled, all entries of the matching lists from the Client Filtering Lists section are copied into the deployed client filtering list and applied to all cluster members. Matching lists are selected by their status (List Status must be Active) and type (List Type must match the value of the Client Filter Type selection).

Once a cluster member receives the deployed client filtering list from the Management Center, it immediately applies the list to all currently connected clients and then uses it for newly connecting clients. Blacklisted clients may connect to another cluster if they are configured to support blue-green deployment. Please see the Blue-Green Deployment and Disaster Recovery section in the Hazelcast IMDG Reference Manual for more information.

If some of the cluster members are not reachable from the Management Center, those members keep using the last client list applied to them.

The deploy action in the Cluster Client Filter Settings section is available by clicking on the Deployed/Deploy Changes button. This button also describes if there were any changes in client filtering lists that would lead to changes in the deployed client filtering list as the result of the deploy (Deploy Changes label), or there were no such changes (Deployed label).

The Client Filtering Lists section allows creation, editing and deletion of the client filtering lists. To create a new client filtering list, you need to click the Add New List button, which will open the Create List form, as shown below. Once you enter all fields and entries for the new list, click the Save button to save your modifications.

Add Client Filtering List

The following formats of list entry values are supported:

  • For the IP Address entry type you can specify IP address (IPv4 or IPv6) with optional range characters (* and -) instead of any byte group. For instance, 10.3.10.* refers to IPs between 10.3.10.0 and 10.3.10.255. The 10.3.10.4-18 refers to IPs between 10.3.10.4 and 10.3.10.18 (4 and 18 included).

  • For the Label entry type you can specify any string with optional wildcard characters (*). For instance, green* refers to any label values that start with the green string.

  • For the Instance Name entry type you can specify any string with optional wildcard characters (*). For instance, *-client refers to any label values that end with the -client string.

To modify an existing client filtering list, you need to click the Edit button, which will open the Edit List form, as shown below.

Edit Client Filtering List

To delete an existing client filtering list, you need to click the Delete button and confirm your action in the opened dialog.

Any modifications made in the Client Filtering Lists section will become available to members only after the deploy action.

13. Monitoring Data Structures

This chapter provides information on how you can monitor the Hazelcast data structures in your cluster.

13.1. Maps

You can see a list of all the maps in your cluster by clicking on the Maps menu item on the left panel. A new page is opened on the right, as shown below.

Map Grid View

You can filter the maps shown and you can also sort the table by clicking on the column headers. Clicking on a map name opens a new page for monitoring that map instance on the right, as shown below.

Monitoring Maps

The below subsections explain the portions of this window.

13.1.1. Map Browser

Use the Map Browser tool to retrieve properties of the entries stored in the selected map. To open the Map Browser tool, click on the Map Browser button, located at the top right of the window. Once opened, the tool appears as a dialog, as shown below.

Map Browser

Once the key and the key’s type are specified and the Browse button is clicked, the key’s properties along with its value are listed.

If you are using a serialization mechanism other than standard Java serialization for storing values in your map, you need to configure the client that Management Center uses for connecting to the cluster (as described here). Note that if you have any custom classes, you need to add the JAR containing them to Management Center’s classpath before starting it as described in Providing an Extra Classpath. In case the value stored in the map is not a String or of another Java primitive type, it is rendered as a JSON value as shown below:

Map Browser with Custom Typed Value

13.1.2. Map Config

Use the Map Config tool to set the selected map’s attributes, such as the backup count, TTL, and eviction policy. To open the Map Config tool, click on the Map Config button, located at the top right of the window. Once opened, the tool appears as a dialog, as shown below.

Map Config Tool

You can change any attribute and click the Update button to save your changes.

13.1.3. Map Monitoring

Besides the Map Browser and Map Config tools, the map monitoring page has monitoring options that are explained below. All of these options perform real-time monitoring.

On top of the page, small charts monitor the entries, operations per second, memory usage, backup size, etc. of the selected map in real-time. The X-axis of all the charts show the current system time. You can select other small monitoring charts using the change window button at the top right of each chart. When you click the button, the monitoring options are listed, as shown below.

Monitoring Options for Map

When you click on a desired monitoring, the chart is loaded with the selected option. To open a chart as a separate dialog, click on the maximize button placed at the top right of each chart. The monitoring charts below are available:

  • Entries: Monitors number of locally owned entries.

  • Memory: Monitors the total memory cost of locally owned entries.

  • Backups: Monitors number of backup entries.

  • Backup Memory: Monitors the total memory cost of backup entries.

  • Hits: Monitors number of hits (reads) of locally owned entries, including those which are no longer in the map (for example, may have been evicted). The number of hits may be inaccurate after a partition is migrated to a new owner member.

  • Locks: Monitors number of locally owned entries that are locked.

  • Operations/s: Monitors the total number of all operations performed on the map per second.

  • Puts/s, Gets/s, Removes/s: These three charts monitor the put, get and remove operations performed on the map per second.

Under these charts are Map Statistics, Map Throughput, Member Near Cache, and Client Near Cache data tables.

Map Statistic Data Table provides statistics distributed over the members, as shown below.

Map Statistic Data Table

From left to right, this table lists the following:

  • Members: IP address and port of the member.

  • Entries: Number of entries owned by the member.

  • Gets: Number of get operations received by the member.

  • Puts: Number of put operations received by the member.

  • Removes: Number of remove operations received by the member.

  • Entry Memory: Memory cost of owned entries in the member.

  • Backups: Number of backup entries held by the member.

  • Backup Memory: Memory cost of backup entries held by the member.

  • Events: Number of events received by the member.

  • Hits: Number of hits (reads) of the entries that are owned by the member, including those which are no longer in the map (for example, may have been evicted). The number of hits may be inaccurate after a partition is migrated to a new owner member.

  • Locks: Number of currently locked entries owned by the member.

  • Dirty Entries: Number of entries that the member owns and are dirty (updated but not persisted yet). In the cases where MapStore is enabled, these are the entries that are put to/removed from the map but not written to/removed from a database yet.

You can ascend or descend the order of the listings by clicking on the column headings.

Map Throughput Data Table provides information about the operations (get, put, remove) performed on each member in the map, as shown below.

Map Throughput Data Table

From left to right, this table lists the following:

  • Members: IP address and port of the member.

  • Puts/s: Number of put operations per second on the member.

  • Gets/s: Number of get operations per second on the member.

  • Removes/s: Number of remove operations per second on the member.

  • Avg Put Latency: Average latency of put operations on the member.

  • Avg Get Latency: Average latency of get operations on the member.

  • Avg Remove Latency: Average latency of remove operations on the member.

  • Max Put Latency: Maximum latency of put operations on the member.

  • Max Get Latency: Maximum latency of get operations on the member.

  • Max Remove Latency: Maximum latency of remove operations on the member.

You can select the time period in the combo box placed on the top right corner of the window, for which the table data will be shown. Available values are Since Beginning, Last Minute, Last 10 Minutes and Last 1 Hour.

To ascend or descend the order of the listings, click on the column headings.

Member Near Cache Data Table provides information about the Member Near Caches, if available, on each member, as shown below.

Member Near Cache Data Table

From left to right, this table lists the following:

  • Members: IP address and port of the member which has Near Caches defined for the maps.

  • Entries: Count of the entries in each Near Cache.

  • Entry Memory: Memory cost of the entries in each Near Cache.

  • Hits: Count of the entries read from the Near Cache.

  • Misses: Count of the entries which cannot be found in the Near Cache when requested to read.

  • Ratio: Hits/Misses ratio.

To ascend or descend the order of the listings, click on the column headings.

Client Near Cache Summary provides summary information related to the Near Cache statistics aggregated for all the clients that have Near Cache enabled for this map. Aggregated statistics are shown for the following periods: 1 minute, 5 minutes, 30 minutes and 60 minutes. Currently, the table shows overall Near Cache effectiveness, calculated as hits/total reads ratio.

You need to enable the statistics for clients to see them here. Please refer to Monitoring Clients for details.
Maps Client Near Cache Summary

Client Near Cache Data Table provides information about the Near Caches statistics, if available, on each client that has Near Cache enabled for this map, as shown below.

You need to enable the statistics for clients to see them here. Please refer to Monitoring Clients for details.
Maps Client Near Cache Data Table

From left to right, this table lists the following:

  • Client Name: Name of the client instance which has Near Cache defined for the map.

  • Client Type: Type of the client.

  • Client Version: Version of the client.

  • Client UUID: Client unique identifier.

  • Evictions: Number of evictions of Near Cache entries owned by the client.

  • Expirations: Number of TTL and max-idle expirations of Near Cache entries owned by the client.

  • Hits: Number of hits (reads) of Near Cache entries owned by the client.

  • Misses: Number of misses of Near Cache entries owned by the client.

  • Effectiveness: Hits/total reads ratio.

  • Owned Entry Count: Number of Near Cache entries owned by the client.

  • Owned Entry Memory Cost: Memory cost of Near Cache entries owned by the client.

To ascend or descend the order of the listings, click on the column headings.

13.2. Caches

You can see a list of all the caches in your cluster by clicking on the Caches menu item on the left panel. A new page is opened on the right, as shown below.

Cache Grid View

You can filter the caches shown and you can also sort the table by clicking on the column headers. Clicking on the cache name opens a new page for monitoring that cache instance on the right, as shown below.

Monitoring Caches

On top of the page, four charts monitor the Gets, Puts, Removals and Evictions in real-time. The X-axis of all the charts show the current system time. To open a chart as a separate dialog, click on the maximize button placed at the top right of each chart.

Under these charts are Cache Statistics, Cache Throughput, and Client Near Cache data tables.

Cache Statistics Data Table provides the selected cache’s statistics distributed over the members, as shown below.

Cache Statistics Data Table

From left to right, this table lists the following in real time:

  • Members: IP address and port of the member.

  • Entries: Number of entries in this cache owned by the member.

  • Gets/Puts/Removals: Number of the get/put/remove operations for this cache received by the member.

  • Hits: Number of the reads performed for this cache’s entries.

  • Misses: Number of the entries which cannot be found in the cache when requested to read.

  • Avg Get/Put Time: Average elapsed time for the get and put operations for the cache on each member.

To ascend or descend the order of the listings, click on the column headings.

Cache Throughput Data Table provides information about the operations (get, put, remove) performed on each member for the selected cache.

Cache Throughput Data Table

From left to right, this table lists the following:

  • IP address and port of each member.

  • Put, get and remove operation rates on each member for this cache.

You can select the period in the combo box placed at the top right corner of the window, for which the table data will be shown. Available values are Since Beginning, Last Minute, Last 10 Minutes and Last 1 Hour.

You can ascend or descend the order of the listings in each column by clicking on column headings.

Client Near Cache Summary provides summary information related to the Near Cache statistics aggregated for all the clients that have Near Cache enabled for this cache. Aggregated statistics are shown for the following periods: 1 minute, 5 minutes, 30 minutes and 60 minutes. Currently, the table shows overall Near Cache effectiveness, calculated as hits/total reads ratio.

You need to enable the statistics for clients to see them here. Please refer to Monitoring Clients for details.
Maps Client Near Cache Summary

Client Near Cache Data Table provides information about the Near Caches statistics, if available, on each client that has Near Cache enabled for this cache, as shown below.

You need to enable the statistics for clients to see them here. Please refer to Monitoring Clients for details.
Caches Client Near Cache Data Table

From left to right, this table lists the following:

  • Client Name: Name of the client instance which has Near Cache enabled for the map.

  • Client Type: Type of the client.

  • Client Version: Version of the client.

  • Client UUID: Client unique identifier.

  • Evictions: Number of evictions of Near Cache entries owned by the client.

  • Expirations: Number of TTL and max-idle expirations of Near Cache entries owned by the client.

  • Hits: Number of hits (reads) of Near Cache entries owned by the client.

  • Misses: Number of misses of Near Cache entries owned by the client.

  • Effectiveness: Hits/total reads ratio.

  • Owned Entry Count: Number of Near Cache entries owned by the client.

  • Owned Entry Memory Cost: Memory cost of Near Cache entries owned by the client.

To ascend or descend the order of the listings, click on the column headings.

You need to enable the statistics for caches to monitor them in the Management Center. Use the <statistics-enabled> element or setStatisticsEnabled() method in declarative or programmatic configuration, respectively, to enable the statistics. Please refer to the JCache Declarative Configuration section for more information.

13.3. Replicated Maps

You can see a list of all the Replicated Maps in your cluster by clicking on the Replicated Maps menu item on the left panel. A new page is opened on the right, as shown below.

Replicated Map Grid View

You can filter the Replicated Maps shown and you can also sort the table by clicking on the column headers. Clicking on a Replicated Map name opens a new page for monitoring that Replicated Map instance on the right, as shown below.

Monitoring Replicated Maps

In this page, you can monitor metrics and also re-configure the selected Replicated Map. All of the statistics are real-time monitoring statistics.

When you click on a desired monitoring, the chart is loaded with the selected option. Also you can open the chart in new window.

  • Entries: Monitors number of entries of the Replicated Map.

  • Operations/s: Monitors number of all operations performed on the Replicated Map per second.

  • Memory: Monitors memory usage of the Replicated Map.

  • Hits: Monitors hit count of the Replicated Map.

  • Puts/s, Gets/s, Removes/s: These three charts monitor the put, get and remove operations performed on the selected Replicated Map per second.

Under these charts are Replicated Map Statistics and Replicated Map Throughput data tables.

Replicated Map Statistics Data Table provides statistics distributed over the members, as shown below.

Replicated Map Statistics Data Table

From left to right, this table lists the following:

  • Members: IP address and port of the member.

  • Entries: Number of entries in this Replicated Map owned by the member.

  • Gets/Puts/Removals: Number of the get/put/remove operations for this Replicated Map received by the member.

  • Entry Memory: Memory cost of the owned entries in the member.

  • Events: Number of the events received by the member.

  • Hits: Number of the reads performed for this Replicated Map’s entries.

Replicated Map Throughput Data Table provides information about operations (get, put, remove) performed on each member in the selected Replicated Map.

Replicated Map Throughput Data Table

From left to right, this table lists the following:

  • IP address and port of each member

  • put, get, and remove operations on each member

  • average put, get, and remove latencies

  • maximum put, get, and remove latencies on each member.

You can select the period from the combo box placed at the top right corner of the window, in which the table data is shown. Available values are Since Beginning, Last Minute, Last 10 Minutes and Last 1 Hour.

To ascend or descend the order of the listings, click on the column headings.

13.4. MultiMaps

You can see a list of all the MultiMaps in your cluster by clicking on the MultiMaps menu item on the left panel. A new page is opened on the right, as shown below.

MultiMap Grid View

You can filter the MultiMaps shown and you can also sort the table by clicking on the column headers. Clicking on a MultiMap name opens a new page for monitoring that MultiMap instance on the right.

MultiMap is a specialized map where you can associate a key with multiple values. This monitoring option is similar to the Maps option: the same monitoring charts and data tables monitor MultiMaps. The differences are that you cannot browse the MultiMaps and re-configure it. Please see the Managing Maps.

13.5. Queues

You can see a list of all the queues in your cluster by clicking on the Queues menu item on the left panel. A new page is opened on the right, as shown below.

Queue Grid View

You can filter the queues shown and you can also sort the table by clicking on the column headers. Clicking on a queue name opens a new page for monitoring that queue instance on the right, as shown below.

Monitoring Queues

On top of the page, small charts monitor the size, offers and polls of the selected queue in real-time. The X-axis of all the charts shows the current system time. To open a chart as a separate dialog, click on the maximize button placed at the top right of each chart. The monitoring charts below are available:

  • Items: Monitors the size of the queue. Y-axis is the entry count.

  • Offers: Monitors the offers sent to the selected queue. Y-axis is the offer count.

  • Polls: Monitors the polls sent to the selected queue. Y-axis is the poll count.

Under these charts are Queue Statistics and Queue Operation Statistics tables.

Queue Statistics table provides item and backup item counts in the queue and age statistics of items and backup items at each member, as shown below.

Queue Statistics

From left to right, this table lists the IP address and port, items and backup items on the queue of each member, and maximum, minimum and average age of items in the queue. The order of the listings in each column can be ascended or descended by clicking on the column headings.

Queue Operation Statistics table provides information about the operations (offers, polls, events) performed on the queues, as shown below.

Queue Operation Statistics

From left to right, this table lists the IP address and port of each member, and counts of offers, rejected offers, polls, poll misses and events.

You can select the period in the combo box placed at the top right corner of the window to show the table data. Available values are Since Beginning, Last Minute, Last 10 Minutes and Last 1 Hour.

Click on the column headings to ascend or descend the order of the listings.

13.6. Topics

You can see a list of all the topics in your cluster by clicking on the Topics menu item on the left panel. A new page is opened on the right, as shown below.

Topic Grid View

You can filter the topics shown and you can also sort the table by clicking on the column headers. Clicking on a topic name opens a new page for monitoring that topic instance on the right, as shown below.

Monitoring Topics

On top of the page, two charts monitor the Publishes and Receives in real-time. They show the published and received message counts of the cluster, the members of which are subscribed to the selected topic. The X-axis of both charts show the current system time. To open a chart as a separate dialog, click on the maximize button placed at the top right of each chart.

Under these charts is the Topic Operation Statistics table. From left to right, this table lists the IP addresses and ports of each member, and counts of the messages published and received per second in real-time. You can select the period in the combo box placed at top right corner of the table to show the table data. The available values are Since Beginning, Last Minute, Last 10 Minutes and Last 1 Hour.

Click on the column heading to ascend or descend the order of the listings.

13.7. Reliable Topics

You can see a list of all the Reliable Topics in your cluster by clicking on the Reliable Topics menu item on the left panel. A new page is opened on the right, as shown below.

Reliable Topic Grid View

You can filter the Reliable Topics shown and you can also sort the table by clicking on the column headers. Clicking on a Reliable Topic name opens a new page for monitoring that Reliable Topic instance on the right, as shown below.

Monitoring Reliable Topics

On top of the page, two charts monitor the Publishes and Receives in real-time. They show the published and received message counts of the cluster, the members of which are subscribed to the selected reliable topic. The X-axis of both charts show the current system time. To open a chart as a separate dialog, click on the maximize button placed at the top right of each chart.

Under these charts is the Reliable Topic Operation Statistics table. From left to right, this table lists the IP addresses and ports of each member, and counts of the messages published and received per second in real-time. You can select the period in the combo box placed at top right corner of the table to show the table data. The available values are Since Beginning, Last Minute, Last 10 Minutes and Last 1 Hour.

Click on the column heading to ascend or descend the order of the listings.

13.8. Executors

You can see a list of all the Executors in your cluster by clicking on the Executors menu item on the left panel. A new page is opened on the right, as shown below.

Executor Grid View

You can filter the Executors shown and you can also sort the table by clicking on the column headers. Clicking on an Executor name opens a new page for monitoring that Executor instance on the right, as shown below.

Monitoring Executors

On top of the page, small charts monitor the pending, started, completed, etc. executors in real-time. The X-axis of all the charts shows the current system time. You can select other small monitoring charts using the change window button placed at the top right of each chart. Click the button to list the monitoring options, as shown below.

Monitoring Options for Executor

When you click on a desired monitoring, the chart loads with the selected option. To open a chart as a separate dialog, click on the maximize button placed at top right of each chart. The below monitoring charts are available:

  • Pending: Monitors the pending executors. Y-axis is the executor count.

  • Started: Monitors the started executors. Y-axis is the executor count.

  • Start Lat. (msec.): Shows the latency when executors are started. Y-axis is the duration in milliseconds.

  • Completed: Monitors the completed executors. Y-axis is the executor count.

  • Comp. Time (msec.): Shows the completion period of executors. Y-axis is the duration in milliseconds.

Under these charts is the Executor Operation Statistics table, as shown below.

Executor Operation Statistics

From left to right, this table lists the IP address and port of members, the counts of pending, started and completed executors per second, and the execution time and average start latency of executors on each member. Click on the column heading to ascend or descend the order of the listings.

13.9. Locks

You can use the scripting feature of the Management Center to monitor the locks in your cluster. See the Scripting section to learn how to use this feature.

You can use the below scripts to retrieve various information about the locks in your cluster.

To find the number of active locks in your cluster, use the following script:

var findLocks = function() {
    var lockstr = '';
    var node = hazelcast.getCluster().getLocalMember();

    var locks = hazelcast.node.nodeEngine.getService('hz:impl:lockService').getAllLocks();
    return "Active Lock Count : " + locks.size();

}

findLocks();

To print the locks in your cluster, use the following script:

var findLocks = function() {
        var lockStr = '';
        var distributedObjects = hazelcast.getDistributedObjects();
        for each(distributedObject in distributedObjects) {
            if(distributedObject.getServiceName().equals("hz:impl:lockService")){
                lockStr += distributedObject.getName() + '\n';
            }

        }
        return lockStr;
}

findLocks();

To force unlock a lock in your cluster, use the following script:

var forceUnlock = function(lockName) {

    hazelcast.getLock(lockName).forceUnlock();
    return 'OK';

}

forceUnlock('your_Lock_Name');

To check if a lock is being hold by a member, use the following script:

var isLocked = function(lockName) {

    var locked = hazelcast.getLock(lockName).isLocked();
    return lockName + ' -> ' + locked;

}

isLocked('your_Lock_Name');

13.10. PN Counters

You can see a list of all the PN counters in your cluster by clicking on the Counters menu item on the left panel. A new page is opened on the right, as shown below.

Counter Grid View

You can filter the counters shown and you can also sort the table by clicking on the column headers. The monitoring data available are:

  • Increment Operations/s: Average number of times the counter was incremented per second during the last timeslice.

  • Decrement Operations/s: Average number of times the counter was decremented per second during the last timeslice.

  • Number of Replicas: Number of member instances that have a state for the counter.

Clicking on a counter name opens a new page for monitoring that specific counter instance, as shown below.

Monitoring Counters

The table can likewise be sorted by clicking the column headers. It shows IP and port of the members that have a state for the specific counter named in the page’s title. The monitoring data available are:

  • Increment Operations/s: Average number of times the counter was incremented on that member per second during the last timeslice

  • Decrement Operations/s: Average number of times the counter was decremented on that member per second during the last timeslice

  • Value: Current value of the counter on that member.

13.11. Flake ID Generators

You can see a list of all Flake ID Generators in your cluster by clicking on the ID Generators menu item on the left panel. A new page is opened on the right, as shown below.

Flake ID Generator Grid View

You can filter the generators shown and you can also sort the table by clicking on the column headers. The monitoring data available are:

  • Avg. Batch Requests: Average count of batch requests coming from all the members to a generator, i.e., total batch requests from all members to a generator divided by the member count for that generator.

  • Avg. Batch Size: Average size of the ID batches created by a generator, i.e., total number of IDs generated (the sum of IDs for all batches) for all members divided by the total count of batch requests coming from all members.

Clicking on a generator name opens a new page for monitoring that specific generator instance, as shown below.

Monitoring Flake ID Generators

The table can likewise be sorted by clicking the column headers. It shows IP and port of the members that have a state for the specific generator named in the page’s title. The monitoring data available are:

  • Batch Requests: Total count of batch requests to a generator by this member.

  • Avg. Batch Size: Average size of the ID batches created for this member, i.e., total number of IDs generated (the sum of IDs for all batches) for this member divided by the total count of batch requests coming from this member.

The operations per second is not the number of new IDs generated or used but the number of ID batches. The batch size is configurable, usually it contains hundreds or thousands of IDs. A client uses all IDs from a batch before a new batch is requested.

14. Monitoring WAN Replication

WAN replication schemes are listed under the WAN Replication menu item on the left. When you click on a scheme, a new page for monitoring the targets which that scheme has appears on the right, as shown below:

Monitoring WAN Replication

In this page, you see the WAN Replication Operations Table for each target which belongs to this scheme. One of the example tables is shown below:

WAN Replication Operations Table
  • Connected: Status of the member connection to the target.

  • Events Published per Second: Number of published events per second. Please see the paragraph below.

  • Average Event Latency: Average latency of sending a record to the target from this member. Please see the paragraph below.

  • Outbound Queue Size: Number of records waiting in the queue to be sent to the target.

  • Action: Pause, stop or resume replication of a member’s records. You can also clear the event queues in a member using the "Clear Queues" action. For instance, if you know that the target cluster is being shut down, decommissioned, put out of use and it will never come back, you may additionally clear the WAN queues to release the consumed heap after the publisher has been switched. Or, when a failure happens and queues are not replicated anymore, you could manually clear the queues using, again the "Clear Queues" action.

  • State: Shows current state of the WAN publisher on a member. See Changing WAN Publisher State for the list of possible WAN publisher states.

Events Published per Second and Average Event Latency are based on the following internal statistics:

  • Total published event count (TBEC): Total number of events that are successfully sent to the target cluster since the start-up of the member.

  • Total latency (TL): Grand total of each event’s waiting time in the queue, including network transmit and receiving ACK from the target.

Each member sends these two statistics to the Management Center at intervals of 3 seconds (update interval). Management Center derives Events Published per Second and Average Event Latency from these statistics as formulated below:

Events Published per Second = (Current TBEC - Previous TBEC) / Update Interval

Average Event Latency = (Current TL - Previous TL) / (Current TBEC - Previous TBEC)

14.1. Changing WAN Publisher State

A WAN publisher can be in one of the following states:

  • REPLICATING (default): State where both enqueuing new events is allowed, enqueued events are replicated to the target cluster.

  • PAUSED: State where new events are enqueued but they are not dequeued. Some events which have been dequeued before the state was switched may still be replicated to the target cluster but further events will not be replicated.

  • STOPPED: State where neither new events are enqueued nor dequeued. As with the PAUSED state, some events might still be replicated after the publisher has switched to this state.

You can change a WAN publisher’s state by clicking the Change State dropdown button on top right hand corner of the WAN Replication Operations Table.

Changing WAN Publisher State

14.2. WAN Sync

You can initiate a synchronization operation on an IMap for a specific target cluster. This operation is useful if two remote clusters lost their synchronization due to WAN queue overflow or in restart scenarios.

Hazelcast provides the following synchronization options:

  1. Default WAN synchronization operation: It sends all the data of an IMap to a target cluster to align the state of target IMap with the source IMap. See here for more information.

  2. WAN synchronization using Merkle trees: To initiate this type of synchronization, you need to configure the cluster members. See the Delta WAN Synchronization section in Hazelcast IMDG Reference Manual for details about configuring them. Make sure you meet all the requirements to use Delta WAN Synchronization and do the configuration on both the source and target clusters.

To initiate WAN Sync, open the WAN Replication menu item on the left and navigate to the Sync tab.

WAN Sync Tab

Click Start button to open the dialog, enter the target details for the sync operation and click Sync to start the operation.

WAN Sync Dialog

You can also use the "All Maps" option in the above dialog if you want to synchronize all the maps in source and target cluster.

You can see the progress of the operation once you initiate it.

WAN Sync Progress

14.3. WAN Consistency Check

You can check if an IMap is in sync with a specific target cluster. Click Check button to open the dialog, enter the target details for the consistency check operation and click Check Consistency to start the operation.

WAN Consistency Check Dialog

You can see the progress of the operation once you initiate it.

WAN Consistency Check Progress
You need to use Merkle trees for WAN synchronization to be able to check for the consistency between two clusters. You need to configure the Merkle trees on both the source and target clusters. If you do not configure it for the source cluster, consistency check is ignored. If it’s enabled for the source cluster but not for the target cluster, all entries are reported as if they need a sync because a sync operation will be a full sync in the absence of Merkle trees.
WAN Consistency Check Ignored

14.4. Add Temporary WAN Replication Configuration

You can add a temporary WAN replication configuration dynamically to a cluster. It is useful for having one-off WAN sync operations. The added configuration has two caveats:

  • It is not persistent, so it does not survive a member restart.

  • It cannot be used as a target for regular WAN replication. It can only be used for WAN sync.

Add Temporary WAN Replication Configuration

See the WAN Replication section in Hazelcast IMDG Reference Manual for details about the fields and their possible values.

After clicking the Add Configuration button, the new WAN replication configuration is added to the cluster. You can see the new configuration when you try to initiate a WAN sync operation as described in the previous section.

15. Scripting

You can use the scripting feature of the Management Center to execute scripts on the cluster. Note that it’s disabled by default and you need to enable it in your cluster’s configuration. Please see the Toggle Scripting Support section in the Hazelcast IMDG Reference Manual for details. Members list shows whether scripting is enabled or disabled for each member.

Member Grid View

To use this feature, click on the Scripting menu item on the left panel. Once selected, the scripting feature opens as shown below.

Scripting

In this window, the left panel is the actual script code editor. The combo box located at the top part of the editor enables you to select a scripting language: currently, JavaScript, Ruby, Groovy and Python languages are supported. You can select the members on which the code will execute from the Members list shown at the right side of the window. After you write your script and press the Execute button, you can see the execution result in the Result part of the window.

To use the scripting languages other than JavaScript on a member, the libraries for those languages should be placed in the classpath of that member. See the Enabling Additional Scripting Languages section for more details.

There are Save and Delete buttons on the top right of the scripting editor. To save your scripts, press the Save button after you type a name for your script into the field next to this button. The scripts you saved are listed in the Saved Scripts part of the window, located at the bottom right of the page. Click on a saved script from this list to execute or edit it. If you want to remove a script that you wrote and saved before, select it from this list and press the Delete button.

In the scripting engine you have a HazelcastInstance bonded to a variable named hazelcast. You can invoke any method that HazelcastInstance has via the hazelcast variable. You can see an example usage for JavaScript below:

var name = hazelcast.getName();
var node = hazelcast.getCluster().getLocalMember();
var employees = hazelcast.getMap("employees");
employees.put("1","John Doe");
employees.get("1"); // will return "John Doe"

15.1. Enabling Additional Scripting Languages

IMDG members use the javax.script.ScriptEngineManager API for the execution of the script. Only the JavaScript script engine is available in most JVMs by default. To use a scripting language other than JavaScript on a member, you need to add the corresponding scripting engine in the classpath of that member. Here is an example of Maven configuration for several scripting engines:

<dependencies>
    <!-- Groovy -->
    <dependency>
        <groupId>org.codehaus.groovy</groupId>
        <artifactId>groovy-jsr223</artifactId>
        <version>2.5.6</version>
    </dependency>
    <!-- Ruby (JRuby) -->
    <dependency>
        <groupId>org.mule.modules</groupId>
        <artifactId>mule-module-scripting-jruby</artifactId>
        <version>3.9.0</version>
    </dependency>
    <dependency>
        <groupId>joda-time</groupId>
        <artifactId>joda-time</artifactId>
        <version>2.10.2</version>
    </dependency>
    <!-- Python (Jython) -->
    <dependency>
        <groupId>org.python</groupId>
        <artifactId>jython-standalone</artifactId>
        <version>2.7.1</version>
    </dependency>
</dependencies>

16. Executing Console Commands

The Management Center has a console feature that enables you to execute commands on the cluster. For example, you can perform puts and gets on a map, after you set the namespace with the command ns <name of your map>. The same is valid for queues, topics and other data structures that can be monitored on the Management Center. To execute your command, type it into the field below the console and press Enter. Type help to see all the commands that you can use.

Open a console window by clicking on the Console button located on the left panel or the one located at the top menu. You can also open it by using ALT + T keyboard shortcut on Windows/Linux and Option + T on macOS. Below is a sample view with the help command executed.

Console

The Management Center sends commands to one of the cluster members; for this, it makes an HTTP request to the REST endpoint on that member. As you can see in the above screenshot, the console screen shows the IP address of the member which receives the console commands. Basically, it connects to the port that member listens to, which is configured on the member side as described here. An example configuration on the member side is shown below:

<hazelcast>
    ...
    <network>
        <port port-count="20" auto-increment="true">5701</port>
    </network>
    ...
</hazelcast>

The direction of this communication is from an ephemeral port number on the Management Center to the port number 5701 (according to the above example) on the member.

There is no configuration option to specify which outbound ports the Management Center will use; it picks on from the ephemeral port pool. Note that, this can create an issue where a firewall is placed between the cluster and Management Center.

17. Cluster Administration

Using the "Administration" menu item, you can change the state of your cluster, shut down it, update your Management Center license and perform Rolling Upgrade or Hot Restart on your cluster.

When you click on the "Administration" menu item, the following page shows up:

Administration Menu
This menu item is available only to admin users.

You can perform the aforementioned administrative tasks using the tabs on this page. Below sections explain each tab.

17.1. Cluster State

The admin user can see and change the cluster state and shut down the cluster using the buttons listed in this page as shown below.

Cluster State Operations

Cluster States:

  • Active: The cluster continues to operate without any restriction. All operations are allowed. This is the default state of a cluster.

  • No Migration: Migrations (partition rebalancing) and backup replications are not allowed. The cluster continues to operate without any restriction. All other operations are allowed.

  • Frozen: New members are not allowed to join, except the members left in this or the Passive state. All other operations except migrations are allowed and the cluster operates without any restriction.

  • Passive: New members are not allowed to join, except the members left in this or the Frozen state. All operations, except the ones marked with AllowedDuringPassiveState, are rejected immediately.

  • In Transition: Shows that the cluster state is in transition. This is a temporary and intermediate state. It is not allowed to set it explicitly.

Changing the Cluster State

Changing Cluster state
  • Click the dropdown menu and choose the state to which you want your cluster to change. A pop-up appears and stays on the screen until the state is successfully changed.

Waiting the State Change

Shutting Down the Cluster

  • Click the Shutdown button. A pop-up appears and stays on the screen until the cluster is successfully shutdown.

Shutdown Cluster

If an exception occurs during the state change or shutdown operation on the cluster, this exception message is shown on the screen as a notification.

17.2. Rolling Upgrade

The admin user can upgrade the cluster version once all members of the cluster have been upgraded to the intended codebase version as described in the Rolling Upgrade Procedure section of the Hazelcast IMDG Reference Manual.

Open the Rolling Upgrade tab to perform a Rolling Upgrade and change the cluster’s version.

RollingUpgradeMenu

Enter the password of the cluster if security is enabled on the cluster (leave it empty if not), and click on the Change Version button.

Once the operation succeeds, you will see the following notification:

UpgradeClusterVersionSuccess

17.3. Hot Restart

Using the Hot Restart tab, you can perform force and partial start of the cluster and see the Hot Restart status of the cluster members. You can also take snapshots of the Hot Restart Store (Hot Backup). When you click on this tab, the following page is shown:

Hot Restart Tab

Below sections explain each operation.

17.3.1. Force Start

Restart process cannot be completed if a member crashes permanently and cannot recover from the failure since it cannot start or it fails to load its own data. In that case, you can force the cluster to clean its persisted data and make a fresh start. This process is called force start.

See the Force Start section in the Hazelcast IMDG Reference Manual for more information on this operation.

To perform a force start on the cluster, click on the Force Start button. A confirmation dialog appears as shown below.

Force Start Confirmation

Once you click on the Force Start button on this dialog, the cluster starts the force start process and the following progress dialog shows up while doing so.

Force Starting

This dialog stays on the screen until the operation is triggered. Once it is done, the success of force start operation is shown as a notice dialog, as shown below.

Force Start Success

If an exception occurs, this exception message is shown on the screen as a notification.

17.3.2. Partial Start

When one or more members fail to start or have incorrect Hot Restart data (stale or corrupted data) or fail to load their Hot Restart data, the cluster becomes incomplete and the restart mechanism cannot proceed. One solution is to use Force Start and make a fresh start with the existing members, as explained above. Another solution is to do a partial start.

Partial start means that the cluster will start with an incomplete set of members. Data belonging to the missing members is assumed lost and the Management Center tries to recover the missing data using the restored backups. For example, if you have minimum two backups configured for all the maps and caches, then a partial start up to two missing members is safe against data loss. If there are more than two missing members or there are maps/caches with less than two backups, then data loss is expected.

See the Partial Start section in the Hazelcast IMDG Reference Manual for more information on this operation and how to enable it.

To perform a partial start on the cluster, click on the Partial Start button. A notice dialog appears as shown below.

Partial Start Triggered

You can also see two fields related to Partial Start operation: "Remaining Data Load Time" and "Remaining Validation Time", as shown in the above screenshot.

  • Remaining Validation Time: When partial start is enabled, Hazelcast can perform a partial start automatically or manually, in case of some members are unable to restart successfully. Partial start proceeds automatically when some members fail to start and join to the cluster in validation-timeout-seconds, which you can configure. After this duration is passed, Hot Restart chooses to perform a partial start with the members present in the cluster. This field, i.e., "Remaining Validation Time" shows how much time is left to the automatic partial start, in seconds. You can always request a manual partial start, by clicking on the Partial Start button, before this duration passes.

  • Remaining Data Load Time: The other situation to decide to perform a partial start is the failures during the data loading phase. When Hazelcast learns the data loading result of all members which have passed the validation step, it automatically performs a partial start with the ones which have successfully restored their Hot Restart data. Note that partial start does not expect every member to succeed in the data loading step. It completes the process when it learns the data loading result for every member and there is at least one member which has successfully restored its Hot Restart data. Relatedly, if it cannot learn the data loading result of all members before data-load-timeout-seconds duration, it proceeds with the ones which have already completed the data loading process. This field, i.e., "Remaining Data Load Time" shows how much time (in seconds) is left for Hazelcast to know at least one member has successfully restored its Hot Restart data and perform an automatic partial start.

See the Configuring Hot Restart section in the Hazelcast IMDG Reference Manual for more information on the configuration elements validation-timeout-seconds and data-load-timeout-seconds mentioned above and how to configure them.
Force and partial start operations can also be performed using the REST API and the script cluster.sh. See the Using REST API for Cluster Management section and Using the Script cluster.sh section in the Hazelcast IMDG Reference Manual.

17.3.3. Hot Backup

During Hot Restart operations, you can take a snapshot of the Hot Restart data at a certain point in time. This is useful when you wish to bring up a new cluster with the same data or parts of the data. The new cluster can then be used to share load with the original cluster, to perform testing, quality assurance or reproduce an issue on the production data.

Note that you must first configure the Hot Backup directory programmatically (using the method setBackupDir()) or declaratively (using the element backup-dir) to be able to take a backup of the Hot Restart data. See the Configuring Hot Backup section in the Hazelcast IMDG Reference Manual.

If the backup directory is configured, you can start to perform the backup by clicking on the Hot Backup button. The Management Center first asks the cluster password as shown in the following dialog.

Hot Backup Ask Cluster Password

Once you entered the password correctly and click on the "Start" button on this dialog, you will see a notification dialog stating that the backup process starts. You can see the progress of the backup operation under the "Last Hot Backup Task Status" part of the page, as shown below.

Hot Backup Progress

17.3.4. Status Information

At the bottom of "Hot Restart" tab, you can see the Hot Restart and Hot Backup statuses of cluster members, as shown below.

Status

You can see the status and progress of your Hot Backup operation under "Last Hot Backup Task Status". It can be IN_PROGRESS and SUCCESS/FAILURE according to the result of the operation.

You can also see the status of Hot Restart operation of your cluster members, under "Hot Restart Status". It can be PENDING and SUCCESSFUL/FAILED according to the result of Hot Restart operation.

17.4. CP Subsystem

CP subsystem management operations require enabled REST API in the IMDG cluster. See the IMDG documentation for more information.

The CP Subsystem tab can be used to monitor overall status of the CP subsystem in the current cluster and perform certain management operations.

Status

17.4.1. Monitoring CP Subsystem

The Status field shows a summary of the current CP subsystem status. It may have one of the following values:

  • CP Subsystem is not supported by this cluster: Shown for IMDG clusters with version prior to 3.12.

  • CP Subsystem is not enabled: Shown if CP subsystem is not enabled for the current cluster.

  • All CP members are accessible: Shown if there are at least the same amount of accessible CP members as the configured CP member count.

  • CP Subsystem warning: one CP member is not accessible: Shown if there is one missing CP member and the minority count in the CP subsystem is greater than 1. For example, this value is shown when there are 6 accessible CP members and the configured count is 7. In this example, the minority count is 3 members and the majority count is 4 members.

  • CP Subsystem alert: multiple CP members are not accessible: Shown if there are multiple missing CP members, but their count is less than the minority.

  • CP Subsystem error: minority of the CP members are not accessible: Shown if the minority of CP members are missing.

  • CP Subsystem error: majority of the CP members are not accessible: Shown if the majority of CP members are missing.

The CP Members (Accessible/Configured) field shows the current count of accessible CP members and the configured CP members count.

You may promote additional members or remove inaccessible CP members, so the total count of members that participate in the CP subsystem may be greater or less than the configured CP member count. As the Status field considers the configured CP member count as the total CP member count, it should be treated only as a basic health indicator for the CP subsystem.

17.4.2. Managing CP Subsystem

You can also use the CP Subsystem tab to start the following management operations.

Promote Member to CP Subsystem

To promote one of the AP members to become a CP member, click on the Promote button. A confirmation dialog appears as shown below.

Promote Member to CP Confirmation

It asks you to choose one of AP members, i.e., one of the members that do not participate in the CP subsystem. Note that lite members are not shown in the dropdown list as lite members do not store data. Once you press the Promote button, the CP subsystem starts the promote operation for the given member.

Remove CP Member

To remove one of the inaccessible CP members from the CP subsystem, click on the Remove button. A confirmation dialog appears as shown below.

Remove CP Member Confirmation

It asks you to choose one of the members that is not connected to the Management Center, but is known by the cluster’s CP subsystem. Once you press the Remove button, the CP subsystem starts the remote operation for the given member.

Restart CP Subsystem

To wipe and restart the whole CP subsystem of the cluster, click on the Restart button. A confirmation dialog appears as shown below.

Restart CP Subsystem Confirmation

Once you press the Restart button, CP subsystem proceeds with the restart operation.

The CP subsystem restart operation is NOT idempotent and multiple invocations can break the whole system! After using this dialog, you must observe the system to see if the restart process is successfully completed or failed before starting this operation again.

18. Using Time Travel

Use the Time Travel toolbar item to check the status of the cluster at a time in the past. When you select it on the toolbar, a small window appears on top of the page, as shown below:

time travel

To see the cluster status in a past time, you should first enable the time travel. Turn on the switch with the "Time Travel" label. It changes to ON after it asks whether to enable the time travel with a dialog. Click on Enable in the dialog to enable it.

Once it is ON, the status of your cluster will be stored on your disk as long as your web server is alive.

Unless you turn on the switch with the "Travel to chosen timestamp" label, you will continue seeing the latest data. When you turn the switch on, you can go back in time using the calendar and check your cluster’s situation at the selected time. All the data structures and members can be monitored as if you are using the Management Center normally (charts and data tables for each data structure and members). It shows the status if time travel has been ON at the selected time in past; otherwise, all the charts and tables are shown as empty.

In the "Travel to chosen timestamp" mode, the graphs do not refresh continuously. You will see data for the selected time. You can press the blue button next to the calendar to see the latest data. Note that this will only show you the latest data and not cause the charts and data tables refresh with latest data continuously. For that, you need to turn off the switch with the "Travel to chosen timestamp" label.

The historical data collected with the time travel feature is stored in a file database on the disk. The data files can be found in the <User‘s Home Directory>/hazelcast-mc directory, e.g., /home/someuser/hazelcast-mc. This directory can be changed using the hazelcast.mc.home property on the server where the Management Center is running.

Time travel data files are created monthly. Their file name format is [cluster-name]-[year][month].db and [cluster-name]-[year][month].lg. Time travel data is kept in the *.db files. The files with the extension lg are temporary files created internally and you do not have to worry about them.

Due to security concerns, time travel can only be used if the cluster name consists of alphanumeric characters, underscores and dashes.

19. Clustered REST

Hazelcast IMDG Enterprise

The Clustered REST API is exposed from the Management Center to allow you to monitor clustered statistics of distributed objects.

19.1. Enabling Clustered REST

To enable Clustered REST on your Management Center, pass the following system property at startup. This property is disabled by default.

-Dhazelcast.mc.rest.enabled=true

19.2. Clustered REST API Root

The entry point for the Clustered REST API is /rest/. This resource does not have any attributes.

All parameters that are used in the REST API URLs, like cluster names and distributed data structure names, must be URL encoded when composing a valid request for Clustered REST. Such parameters are marked in braces ({ and }) in the URL description for each endpoint. As an example, name.with/special@chars parameter value would be encoded as name.with%2Fspecial%40chars.
All endpoints return HTTP status code 404 if no data about a cluster, member, client or data structure can be found in the Management Center.

19.2.1. Retrieve Management Center License Expiration Time

This endpoint returns the expiration time in milliseconds (since epoch) of the license key assigned for the Management Center. Returns -1 if no license is assigned.

  • Request Type: GET

  • URL: /rest/license

  • Request:

    curl http://localhost:8080/rest/license
  • Response: 200 (application/json)

  • Body:

    {
      "licenseExpirationTime": 4099755599515
    }

19.3. Clusters Resource

This resource returns a list of clusters that are connected to the Management Center.

19.3.1. Retrieve Clusters

  • Request Type: GET

  • URL: /rest/clusters

  • Request:

    curl http://localhost:8080/rest/clusters
  • Response: 200 (application/json)

  • Body:

    ["dev", "qa"]

19.4. Cluster Resource

This resource returns information related to the provided cluster name.

19.4.1. Retrieve Cluster Information

This endpoint returns address of the oldest cluster member and the expiration time in milliseconds (since epoch) of the license key assigned for the cluster. Returns -1 for license expiration time if no license is assigned.

  • Request Type: GET

  • URL: /rest/clusters/{clustername}

  • Request:

    curl http://localhost:8080/rest/clusters/dev/
  • Response: 200 (application/json)

  • Body:

    {
      "masterAddress": "192.168.2.78:5701",
      "licenseExpirationTime": 4099755599515
    }

19.5. Members Resource

This resource returns a list of the members belonging to the provided clusters.

19.5.1. Retrieve Members [GET] [/rest/clusters/{clustername}/members]

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/members

  • Request:

    curl http://localhost:8080/rest/clusters/dev/members
  • Response: 200 (application/json)

  • Body:

    [
      "192.168.2.78:5701",
      "192.168.2.78:5702",
      "192.168.2.78:5703",
      "192.168.2.78:5704"
    ]

19.6. Member Resource

This resource returns information related to the provided member.

19.6.1. Retrieve Member Information

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/members/{member}

  • Request:

    curl http://localhost:8080/rest/clusters/dev/members/192.168.2.78:5701
  • Response: 200 (application/json)

  • Body:

    {
      "cluster": "dev",
      "name": "192.168.2.78:5701",
      "uuid": "11adba52-e19d-4407-a9e9-e0a271cef14a",
      "cpMemberUuid": "f5a8f8a4-f278-4a13-a23e-5accf5b02f42",
      "maxMemory": 129957888,
      "ownedPartitionCount": 68,
      "usedMemory": 60688784,
      "freeMemory": 24311408,
      "totalMemory": 85000192,
      "connectedClientCount": 1,
      "master": true
    }

19.6.2. Retrieve Connection Manager Information

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/members/{member}/connectionManager

  • Request:

    curl http://localhost:8080/rest/clusters/dev/members/192.168.2.78:5701/connectionManager
  • Response: 200 (application/json)

  • Body:

    {
      "clientConnectionCount": 2,
      "activeConnectionCount": 5,
      "connectionCount": 5
    }

19.6.3. Retrieve Operation Service Information

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/members/{member}/operationService

  • Request:

    curl http://localhost:8080/rest/clusters/dev/members/192.168.2.78:5701/operationService
  • Response: 200 (application/json)

  • Body:

    {
      "responseQueueSize": 0,
      "operationExecutorQueueSize": 0,
      "runningOperationsCount": 0,
      "remoteOperationCount": 1,
      "executedOperationCount": 461139,
      "operationThreadCount": 8
    }

19.6.4. Retrieve Event Service Information

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/members/{member}/eventService

  • Request:

    curl http://localhost:8080/rest/clusters/dev/members/192.168.2.78:5701/eventService
  • Response: 200 (application/json)

  • Body:

    {
      "eventThreadCount": 5,
      "eventQueueCapacity": 1000000,
      "eventQueueSize": 0
    }

19.6.5. Retrieve Partition Service Information

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/members/{member}/partitionService

  • Request:

    curl http://localhost:8080/rest/clusters/dev/members/192.168.2.78:5701/partitionService
  • Response: 200 (application/json)

  • Body:

    {
      "partitionCount": 271,
      "activePartitionCount": 68
    }

19.6.6. Retrieve Proxy Service Information

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/members/{member}/proxyService

  • Request:

    curl http://localhost:8080/rest/clusters/dev/members/192.168.2.78:5701/proxyService
  • Response: 200 (application/json)

  • Body:

    {
      "proxyCount": 8
    }

19.6.7. Retrieve All Managed Executors

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/members/{member}/managedExecutors

  • Request:

    curl http://localhost:8080/rest/clusters/dev/members/192.168.2.78:5701/managedExecutors
  • Response: 200 (application/json)

  • Body:

    ["hz:system", "hz:scheduled", "hz:client", "hz:query", "hz:io", "hz:async"]

19.6.8. Retrieve a Managed Executor

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/members/{member}/managedExecutors/{managedExecutor}

  • Request:

    curl http://localhost:8080/rest/clusters/dev/members/192.168.2.78:5701
              /managedExecutors/hz:system
  • Response: 200 (application/json)

  • Body:

    {
      "name": "hz:system",
      "queueSize": 0,
      "poolSize": 0,
      "remainingQueueCapacity": 2147483647,
      "maximumPoolSize": 4,
      "completedTaskCount": 12,
      "terminated": false
    }

19.7. Client Endpoints Resource

This resource returns a list of the client endpoints belonging to the provided cluster. Consider using the newly added Client Statistics Resource as it contains more detailed information about the clients.

19.7.1. Retrieve List of Client Endpoints

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/clients

  • Request:

    curl http://localhost:8080/rest/clusters/dev/clients
  • Response: 200 (application/json)

  • Body:

    ["192.168.2.78:61708"]

19.7.2. Retrieve Client Endpoint Information

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/clients/{client}

  • Request:

    curl http://localhost:8080/rest/clusters/dev/clients/192.168.2.78:61708
  • Response: 200 (application/json)

  • Body:

    {
      "uuid": "6fae7af6-7a7c-4fa5-b165-cde24cf070f5",
      "address": "192.168.2.78:61708",
      "clientType": "JAVA",
      "name": "hz.client_1",
      "labels": [
        "label1"
      ],
      "ipAddress": "192.168.2.78",
      "canonicalHostName": "localhost"
    }

19.8. Maps Resource

This resource returns a list of maps belonging to the provided cluster.

19.8.1. Retrieve List of Maps

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/maps

  • Request:

    curl http://localhost:8080/rest/clusters/dev/maps
  • Response: 200 (application/json)

  • Body:

    ["customers", "orders"]

19.8.2. Retrieve Map Information

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/maps/{mapName}

  • Request:

    curl http://localhost:8080/rest/clusters/dev/maps/customers
  • Response: 200 (application/json)

  • Body:

    {
      "cluster": "dev",
      "name": "customers",
      "ownedEntryCount": 5085,
      "backupEntryCount": 5076,
      "ownedEntryMemoryCost": 833940,
      "backupEntryMemoryCost": 832464,
      "heapCost": 1666668,
      "lockedEntryCount": 2,
      "dirtyEntryCount": 0,
      "hits": 602,
      "lastAccessTime": 1532689094579,
      "lastUpdateTime": 1532689094576,
      "creationTime": 1532688789256,
      "putOperationCount": 5229,
      "getOperationCount": 2162,
      "removeOperationCount": 150,
      "otherOperationCount": 3687,
      "events": 10661,
      "maxPutLatency": 48,
      "maxGetLatency": 35,
      "maxRemoveLatency": 18034,
      "averagePutLatency": 0.5674125071715433,
      "averageGetLatency": 0.2479185938945421,
      "averageRemoveLatency": 5877.986666666667
    }

19.9. MultiMaps Resource

This resource returns a list of multimaps belonging to the provided cluster.

19.9.1. Retrieve List of MultiMaps

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/multimaps

  • Request:

    curl http://localhost:8080/rest/clusters/dev/multimaps
  • Response: 200 (application/json)

  • Body:

    ["customerAddresses"]

19.9.2. Retrieve MultiMap Information

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/multimaps/{multimapname}

  • Request:

    curl http://localhost:8080/rest/clusters/dev/multimaps/customerAddresses
  • Response: 200 (application/json)

  • Body:

    {
      "cluster": "dev",
      "name": "customerAddresses",
      "ownedEntryCount": 4862,
      "backupEntryCount": 4860,
      "ownedEntryMemoryCost": 0,
      "backupEntryMemoryCost": 0,
      "heapCost": 0,
      "lockedEntryCount": 1,
      "dirtyEntryCount": 0,
      "hits": 22,
      "lastAccessTime": 1532689253314,
      "lastUpdateTime": 1532689252591,
      "creationTime": 1532688790593,
      "putOperationCount": 5125,
      "getOperationCount": 931,
      "removeOperationCount": 216,
      "otherOperationCount": 373570,
      "events": 0,
      "maxPutLatency": 8,
      "maxGetLatency": 1,
      "maxRemoveLatency": 18001,
      "averagePutLatency": 0.3758048780487805,
      "averageGetLatency": 0.11170784103114931,
      "averageRemoveLatency": 1638.8472222222222
    }

19.10. ReplicatedMaps Resource

This resource returns a list of replicated maps belonging to the provided cluster.

19.10.1. Retrieve List of ReplicatedMaps

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/replicatedmaps

  • Request:

    curl http://localhost:8080/rest/clusters/dev/replicatedmaps
  • Response: 200 (application/json)

  • Body:

    ["replicated-map-1"]

19.10.2. Retrieve ReplicatedMap Information

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/replicatedmaps/{replicatedmapname}

  • Request:

    curl http://localhost:8080/rest/clusters/dev/replicatedmaps/replicated-map-1
  • Response: 200 (application/json)

  • Body:

    {
      "cluster": "dev",
      "name": "replicated-map-1",
      "ownedEntryCount": 10955,
      "ownedEntryMemoryCost": 394380,
      "hits": 15,
      "lastAccessTime": 1532689312581,
      "lastUpdateTime": 1532689312581,
      "creationTime": 1532688789493,
      "putOperationCount": 11561,
      "getOperationCount": 1051,
      "removeOperationCount": 522,
      "otherOperationCount": 355552,
      "events": 6024,
      "maxPutLatency": 1,
      "maxGetLatency": 1,
      "maxRemoveLatency": 1,
      "averagePutLatency": 0.006400830377994983,
      "averageGetLatency": 0.012369172216936251,
      "averageRemoveLatency": 0.011494252873563218
    }

19.11. Queues Resource

This resource returns a list of queues belonging to the provided cluster.

19.11.1. Retrieve List of Queues

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/queues

  • Request:

    curl http://localhost:8080/rest/clusters/dev/queues
  • Response: 200 (application/json)

  • Body:

    ["messages"]

19.11.2. Retrieve Queue Information

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/queues/{queueName}

  • Request:

    curl http://localhost:8080/rest/clusters/dev/queues/messages
  • Response: 200 (application/json)

  • Body:

    {
      "cluster": "dev",
      "name": "messages",
      "ownedItemCount": 55408,
      "backupItemCount": 55408,
      "minAge": 0,
      "maxAge": 0,
      "averageAge": 0,
      "numberOfOffers": 55408,
      "numberOfRejectedOffers": 0,
      "numberOfPolls": 0,
      "numberOfEmptyPolls": 0,
      "numberOfOtherOperations": 0,
      "numberOfEvents": 0,
      "creationTime": 1403602694196
    }

19.12. Topics Resource

This resource returns a list of topics belonging to the provided cluster.

19.12.1. Retrieve List of Topics

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/topics

  • Request:

    curl http://localhost:8080/rest/clusters/dev/topics
  • Response: 200 (application/json)

  • Body:

    ["news"]

19.12.2. Retrieve Topic Information

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/topics/{topicName}

  • Request:

    curl http://localhost:8080/rest/clusters/dev/topics/news
  • Response: 200 (application/json)

  • Body:

    {
      "cluster": "dev",
      "name": "news",
      "numberOfPublishes": 56370,
      "totalReceivedMessages": 56370,
      "creationTime": 1403602693411
    }

19.13. Executors Resource

This resource returns a list of executors belonging to the provided cluster.

19.13.1. Retrieve List of Executors

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/executors

  • Request:

    curl http://localhost:8080/rest/clusters/dev/executors
  • Response: 200 (application/json)

  • Body:

    ["order-executor"]

19.13.2. Retrieve Executor Information [GET] [/rest/clusters/{clustername}/executors/{executorName}]

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/executors/{executorName}

  • Request:

    curl http://localhost:8080/rest/clusters/dev/executors/order-executor
  • Response: 200 (application/json)

  • Body:

    {
      "cluster": "dev",
      "name": "order-executor",
      "creationTime": 1403602694196,
      "pendingTaskCount": 0,
      "startedTaskCount": 1241,
      "completedTaskCount": 1241,
      "cancelledTaskCount": 0
    }

19.14. Client Statistics Resource

This resource returns a list of clients belonging to the provided cluster.

19.14.1. Retrieve List of Client UUIDs

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/clientStats

  • Request:

    curl http://localhost:8080/rest/clusters/dev/clientStats
  • Response: 200 (application/json)

  • Body:

    [
      "f3b1e0e9-ea67-41b2-aba5-ea7480f02a93",
      "cebf4dc9-852c-4605-a181-ffe1cca371a4",
      "2371eed5-26e0-4470-92c1-41ea17110ef6",
      "139990b3-fbc0-43a8-9c12-be53913333f7",
      "d0364a1e-8665-46a8-af1d-be1af5580d07",
      "7f337f8a-3538-4b5c-8ffc-9d4ae459e956",
      "6ef9b6e5-5add-40d9-9319-ce502f55b5fc",
      "fead3a99-19de-431c-9dd0-d6ecc4a4b9c8",
      "e788e04e-2ded-4992-9d76-52c1973216e5",
      "654fc9fb-c5c1-48a0-9b69-0c129fce860f"
    ]

19.14.2. Retrieve Detailed Client Statistics [GET] [/rest/clusters/{clustername}/clientStats/{clientUuid}]

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/clientStats/{clientUuid}

  • Request:

    curl http://localhost:8080/rest/clusters/dev/clientStats/2371eed5-26e0-4470-92c1-41ea17110ef6
  • Response: 200 (application/json)

  • Body:

    {
      "type": "JAVA",
      "name": "hz.client_7",
      "address": "127.0.0.1",
      "clusterConnectionTimestamp": 1507874427419,
      "enterprise": true,
      "lastStatisticsCollectionTime": 1507881309434,
      "osStats": {
        "committedVirtualMemorySize": 12976173056,
        "freePhysicalMemorySize": 3615662080,
        "freeSwapSpaceSize": 8447324160,
        "maxFileDescriptorCount": 1000000,
        "openFileDescriptorCount": 191,
        "processCpuTime": 252980000000,
        "systemLoadAverage": 83.0,
        "totalPhysicalMemorySize": 16756101120,
        "totalSwapSpaceSize": 8447324160
      },
      "runtimeStats": {
        "availableProcessors": 12,
        "freeMemory": 135665432,
        "maxMemory": 3724541952,
        "totalMemory": 361234432,
        "uptime": 6894992,
        "usedMemory": 225569000
      },
      "nearCacheStats": {
        "CACHE": {
          "a-cache": {
            "creationTime": 1507874429719,
            "evictions": 0,
            "hits": 0,
            "misses": 50,
            "ownedEntryCount": 0,
            "expirations": 0,
            "ownedEntryMemoryCost": 0,
            "lastPersistenceDuration": 0,
            "lastPersistenceKeyCount": 0,
            "lastPersistenceTime": 0,
            "lastPersistenceWrittenBytes": 0,
            "lastPersistenceFailure": ""
          },
          "b.cache": {
            "creationTime": 1507874429973,
            "evictions": 0,
            "hits": 0,
            "misses": 50,
            "ownedEntryCount": 0,
            "expirations": 0,
            "ownedEntryMemoryCost": 0,
            "lastPersistenceDuration": 0,
            "lastPersistenceKeyCount": 0,
            "lastPersistenceTime": 0,
            "lastPersistenceWrittenBytes": 0,
            "lastPersistenceFailure": ""
          }
        },
        "MAP": {
          "other,map": {
            "creationTime": 1507874428638,
            "evictions": 0,
            "hits": 100,
            "misses": 50,
            "ownedEntryCount": 0,
            "expirations": 0,
            "ownedEntryMemoryCost": 0,
            "lastPersistenceDuration": 0,
            "lastPersistenceKeyCount": 0,
            "lastPersistenceTime": 0,
            "lastPersistenceWrittenBytes": 0,
            "lastPersistenceFailure": ""
          },
          "employee-map": {
            "creationTime": 1507874427959,
            "evictions": 0,
            "hits": 100,
            "misses": 50,
            "ownedEntryCount": 0,
            "expirations": 0,
            "ownedEntryMemoryCost": 0,
            "lastPersistenceDuration": 0,
            "lastPersistenceKeyCount": 0,
            "lastPersistenceTime": 0,
            "lastPersistenceWrittenBytes": 0,
            "lastPersistenceFailure": ""
          }
        }
      },
      "userExecutorQueueSize": 0,
      "memberConnection": "ALL",
      "version": "UNKNOWN"
    }

19.14.3. Retrieve WAN Publisher Statistics [GET] [/rest/clusters/{clustername}/wanStats/{wanReplication}/publishers/{publisher}]

  • Request Type: GET

  • URL: /rest/clusters/{clustername}/wanStats/{wanReplication}/publishers/{publisher}

  • Request:

    curl http://localhost:8080/rest/clusters/dev/wanStats/devWanConfig/publishers/devPublisher
  • Response: 200 (application/json)

  • Body:

    {
      "cluster": "dev",
      "configName": "devWanConfig",
      "publisherId": "devPublisher",
      "totalPublishedEventCount": 1023,
      "outboundQueueSize": 10,
      "maxOutboundQueueSize": 5,
      "time": 1577696465000
    }

20. Clustered JMX

Hazelcast IMDG Enterprise

Clustered JMX via Management Center allows you to monitor clustered statistics of distributed objects from a JMX interface.

20.1. Configuring Clustered JMX

In order to configure Clustered JMX, use the following command line parameters for your Management Center deployment.

  • -Dhazelcast.mc.jmx.enabled=true (default is false)

  • -Dhazelcast.mc.jmx.port=9000 (optional, default is 9999)

  • -Dcom.sun.management.jmxremote.ssl=false

  • -Dhazelcast.mc.jmx.rmi.port=9001 (optional, default is 9998)

  • -Dhazelcast.mc.jmx.host=localhost (optional, default is server’s host name)

With embedded Jetty, you do not need to deploy your Management Center application to any container or application server.

You can start the Management Center application with Clustered JMX enabled as shown below.

java -Dhazelcast.mc.jmx.enabled=true
-Dhazelcast.mc.jmx.port=9999
-Dcom.sun.management.jmxremote.ssl=false -jar hazelcast-management-center-4.0.war

Once the Management Center starts, you should see a log similar to the one below:

INFO: Management Center 3.3
Jun 05, 2014 11:55:32 AM com.hazelcast.webmonitor.service.jmx.impl.JMXService
INFO: Starting Management Center JMX Service on port :9999

You should be able to connect to the Clustered JMX interface using the address localhost:9999.

You can use jconsole or any other JMX client to monitor your Hazelcast IMDG cluster. As an example, below is the jconsole screenshot of the Clustered JMX hierarchy.

JMX

20.1.1. Enabling TLS/SSL for Clustered JMX

By default, Clustered JMX is served unencrypted. To enable TLS/SSL for Clustered JMX, use the following command line parameters for your Management Center deployment:

  • -Dhazelcast.mc.jmx.ssl=true (default is false)

  • -Dhazelcast.mc.jmx.ssl.keyStore=path to your keystore

  • -Dhazelcast.mc.jmx.ssl.keyStorePassword=password for your keystore

The following is an example on how to start the Management Center with a TLS/SSL enabled Clustered JMX service on port 65432:

java -Dhazelcast.mc.jmx.enabled=true
-Dhazelcast.mc.jmx.port=65432
-Dhazelcast.mc.jmx.ssl=true
-Dhazelcast.mc.jmx.ssl.keyStore=/some/dir/selfsigned.jks
-Dhazelcast.mc.jmx.ssl.keyStorePassword=yourpassword -jar hazelcast-management-center-4.0.war
You can encrypt the keystore password and pass it as a command line argument in encrypted form for improved security. See the Variable Replacers section for more information.

Then, you can use the following command to connect to the Clustered JMX service using JConsole with the address localhost:65432:

jconsole -J-Djavax.net.ssl.trustStore=/some/dir/selftrusted.ts -J-Djavax.net.ssl.trustStorePassword=trustpass
Additional TLS/SSL Configuration Options

The following are some additional command line arguments that you can use to configure TLS/SSL for clustered JMX:

  • -Dhazelcast.mc.jmx.ssl.keyStoreType: Type of the keystore. Its default value is JKS.

  • -Dhazelcast.mc.jmx.ssl.keyManagerAlgorithm: Name of the algorithm based on which the authentication keys are provided. The system default is used if none is provided. You can find out the default by calling the javax.net.ssl.KeyManagerFactory#getDefaultAlgorithm method.

20.2. Clustered JMX API

The management beans are exposed with the following object name format:

ManagementCenter[`*cluster name*`]:type=<`*object type*`>,name=<`*object name*`>,member="<`*cluster member IP address*`>"

The object name starts with the ManagementCenter prefix. Then it has the cluster name in brackets followed by a colon. After that, type, name and member attributes follow, each separated with a comma.

  • type is the type of object. Values are Clients, Executors, Maps, Members, MultiMaps, Queues, Counters, Services, and Topics.

  • name is the name of object.

  • member is the member address of object (only required if the statistics are local to the member).

A sample bean is shown below.

ManagementCenter[dev]:type=Services,name=OperationService,member="192.168.2.79:5701"

Here is the list of attributes that are exposed from the Clustered JMX interface.

  • ManagementCenter

  • ManagementCenter

    • LicenseExpirationTime

    • Clusters

  • ManagementCenter[<ClusterName>]

  • <ClusterName>

    • MasterAddress

    • LicenseExpirationTime

  • ClientStats

    • <Client UUID>

      • HeapUsedMemory

      • HeapFreeMemory

      • HeapMaxMemory

      • HeapTotalMemory

      • ClientName

      • AvailableProcessors

      • Uptime

      • Enterprise

      • MemberConnection

      • ClusterConnectionTimestamp

      • LastStatisticsCollectionTime

      • UserExecutorQueueSize

      • CommittedVirtualMemorySize

      • FreePhysicalMemorySize

      • FreeSwapSpaceSize

      • MaxFileDescriptorCount

      • OpenFileDescriptorCount

      • ProcessCpuTime

      • SystemLoadAverage

      • TotalPhysicalMemorySize

      • TotalSwapSpaceSize

      • Version

      • Address

      • Type

      • CACHE

        • <Cache Name>

          • Evictions

          • Expirations

          • Hits

          • Misses

          • OwnedEntryCount

          • OwnedEntryMemoryCost

          • LastPersistenceDuration

          • LastPersistenceKeyCount

          • LastPersistenceTime

          • LastPersistenceWrittenBytes

          • LastPersistenceFailure

          • CreationTime

      • MAP

        • <Map Name>

          • Evictions

          • Expirations

          • Hits

          • Misses

          • OwnedEntryCount

          • OwnedEntryMemoryCost

          • LastPersistenceDuration

          • LastPersistenceKeyCount

          • LastPersistenceTime

          • LastPersistenceWrittenBytes

          • LastPersistenceFailure

          • CreationTime

  • Clients

    • <Client Address>

      • Address

      • CanonicalHostName

      • ClientName

      • ClientType

      • IpAddress

      • Labels

      • Uuid

  • Executors

    • <Executor Name>

      • Cluster

      • Name

      • StartedTaskCount

      • CompletedTaskCount

      • CancelledTaskCount

      • PendingTaskCount

  • Maps

    • <Map Name>

      • Cluster

      • Name

      • BackupEntryCount

      • BackupEntryMemoryCost

      • CreationTime

      • DirtyEntryCount

      • Events

      • GetOperationCount

      • HeapCost

      • Hits

      • LastAccessTime

      • LastUpdateTime

      • LockedEntryCount

      • MaxGetLatency

      • MaxPutLatency

      • MaxRemoveLatency

      • OtherOperationCount

      • OwnedEntryCount

      • PutOperationCount

      • RemoveOperationCount

      • AverageGetLatency

      • AveragePutLatency

      • AverageRemoveLatency

  • ReplicatedMaps

    • <Replicated Map Name>

      • Cluster

      • Name

      • BackupEntryCount

      • BackupEntryMemoryCost

      • CreationTime

      • DirtyEntryCount

      • Events

      • GetOperationCount

      • HeapCost

      • Hits

      • LastAccessTime

      • LastUpdateTime

      • LockedEntryCount

      • MaxGetLatency

      • MaxPutLatency

      • MaxRemoveLatency

      • OtherOperationCount

      • OwnedEntryCount

      • PutOperationCount

      • RemoveOperationCount

      • AverageGetLatency

      • AveragePutLatency

      • AverageRemoveLatency

  • Members

    • <Member Address>

      • Uuid

      • CpMemberUuid

      • ConnectedClientCount

      • HeapFreeMemory

      • HeapMaxMemory

      • HeapTotalMemory

      • HeapUsedMemory

      • IsMaster

      • OwnedPartitionCount

  • MultiMaps

    • <MultiMap Name>

      • Cluster

      • Name

      • BackupEntryCount

      • BackupEntryMemoryCost

      • CreationTime

      • DirtyEntryCount

      • Events

      • GetOperationCount

      • HeapCost

      • Hits

      • LastAccessTime

      • LastUpdateTime

      • LockedEntryCount

      • MaxGetLatency

      • MaxPutLatency

      • MaxRemoveLatency

      • OtherOperationCount

      • OwnedEntryCount

      • PutOperationCount

      • RemoveOperationCount

      • AverageGetLatency

      • AveragePutLatency

      • AverageRemoveLatency

  • Queues

    • <Queue Name>

      • Cluster

      • Name

      • MinAge

      • MaxAge

      • AverageAge

      • OwnedItemCount

      • BackupItemCount

      • OfferOperationCount

      • OtherOperationsCount

      • PollOperationCount

      • RejectedOfferOperationCount

      • EmptyPollOperationCount

      • EventOperationCount

      • CreationTime

  • Counters

    • <Counter Name>

      • Cluster

      • Name

      • ReplicaCount

      • Time

      • OpsPerSecInc (for each member)

      • OpsPerSecDec (for each member)

      • Value (for each member)

  • Services

    • ConnectionManager

      • ActiveConnectionCount

      • ClientConnectionCount

      • ConnectionCount

    • EventService

      • EventQueueCapacity

      • EventQueueSize

      • EventThreadCount

    • OperationService

      • ExecutedOperationCount

      • OperationExecutorQueueSize

      • OperationThreadCount

      • RemoteOperationCount

      • ResponseQueueSize

      • RunningOperationsCount

    • PartitionService

      • ActivePartitionCount

      • PartitionCount

    • ProxyService

      • ProxyCount

    • ManagedExecutor[hz::async]

      • Name

      • CompletedTaskCount

      • MaximumPoolSize

      • PoolSize

      • QueueSize

      • RemainingQueueCapacity

      • Terminated

    • ManagedExecutor[hz::client]

      • Name

      • CompletedTaskCount

      • MaximumPoolSize

      • PoolSize

      • QueueSize

      • RemainingQueueCapacity

      • Terminated

    • ManagedExecutor[hz::global-operation]

      • Name

      • CompletedTaskCount

      • MaximumPoolSize

      • PoolSize

      • QueueSize

      • RemainingQueueCapacity

      • Terminated

    • ManagedExecutor[hz::io]

      • Name

      • CompletedTaskCount

      • MaximumPoolSize

      • PoolSize

      • QueueSize

      • RemainingQueueCapacity

      • Terminated

    • ManagedExecutor[hz::query]

      • Name

      • CompletedTaskCount

      • MaximumPoolSize

      • PoolSize

      • QueueSize

      • RemainingQueueCapacity

      • Terminated

    • ManagedExecutor[hz::scheduled]

      • Name

      • CompletedTaskCount

      • MaximumPoolSize

      • PoolSize

      • QueueSize

      • RemainingQueueCapacity

      • Terminated

    • ManagedExecutor[hz::system]

      • Name

      • CompletedTaskCount

      • MaximumPoolSize

      • PoolSize

      • QueueSize

      • RemainingQueueCapacity

      • Terminated

  • Topics

    • <Topic Name>

      • Cluster

      • Name

      • CreationTime

      • PublishOperationCount

      • ReceiveOperationCount

  • FlakeIdGenerators

    • <Generator Name>

      • Cluster

      • Name

      • Time

      • OpsPerSec (per member)

  • WanConfigs

    • <Wan Replication Config>[<Publisher ID>]

      • Cluster

      • ConfigName

      • PublisherId

      • Time

      • OutboundQueueSize

      • MaxOutboundQueueSize

      • TotalPublishedEventCount

20.3. Integrating with New Relic

Use the Clustered JMX interface to integrate the Hazelcast Management Center with New Relic. To perform this integration, attach the New Relic Java agent and provide an extension file that describes which metrics will be sent to New Relic.

See Custom JMX instrumentation by YAML on the New Relic webpage.

The following is an example Map monitoring .yml file for New Relic:

name: Clustered JMX
version: 1.0
enabled: true

jmx:
- object_name: ManagementCenter[clustername]:type=Maps,name=mapname
  metrics:
  - attributes: PutOperationCount, GetOperationCount, RemoveOperationCount, Hits, BackupEntryCount, OwnedEntryCount, LastAccessTime, LastUpdateTime
  - type: simple
- object_name: ManagementCenter[clustername]:type=Members,name="member address in double quotes"
  metrics:
  - attributes: OwnedPartitionCount
  - type: simple

Put the .yml file in the extensions directory in your New Relic installation. If an extensions directory does not exist there, create one.

After you set your extension, attach the New Relic Java agent and start the Management Center as shown below.

java -javaagent:/path/to/newrelic.jar -Dhazelcast.mc.jmx.enabled=true\
    -Dhazelcast.mc.jmx.port=9999 -jar hazelcast-management-center-4.0.war

If your logging level is set to FINER, you should see the log listing in the file newrelic_agent.log, which is located in the logs directory in your New Relic installation. The following is an example log listing:

Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINE:
    JMX Service : querying MBeans (1)
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER:
    JMX Service : MBeans query ManagementCenter[dev]:type=Members,
    name="192.168.2.79:5701", matches 1
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER:
    Recording JMX metric OwnedPartitionCount : 68
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER:
    JMX Service : MBeans query ManagementCenter[dev]:type=Maps,name=orders,
    matches 1
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER:
    Recording JMX metric Hits : 46,593
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER:
    Recording JMX metric BackupEntryCount : 1,100
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER:
    Recording JMX metric OwnedEntryCount : 1,100
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER:
    Recording JMX metric RemoveOperationCount : 0
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER:
    Recording JMX metric PutOperationCount : 118,962
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER:
    Recording JMX metric GetOperationCount : 0
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER:
    Recording JMX metric LastUpdateTime : 1,401,962,426,811
Jun 5, 2014 14:18:43 +0300 [72696 62] com.newrelic.agent.jmx.JmxService FINER:
    Recording JMX metric LastAccessTime : 1,401,962,426,811

Then you can navigate to your New Relic account and create Custom Dashboards. See Creating custom dashboards.

While you are creating the dashboard, you should see the metrics that you are sending to New Relic from the Management Center in the Metrics section under the JMX directory.

20.4. Integrating with AppDynamics

Use the Clustered JMX interface to integrate the Hazelcast Management Center with AppDynamics. To perform this integration, attach the AppDynamics Java agent to the Management Center.

For agent installation, see the Install the App Agent for Java page.

For monitoring on AppDynamics, see the Using AppDynamics for JMX Monitoring page.

After installing AppDynamics agent, you can start the Management Center as shown below:

java -javaagent:/path/to/javaagent.jar \
     -Dhazelcast.mc.jmx.enabled=true \
     -Dhazelcast.mc.jmx.port=9999 -jar hazelcast-management-center-4.0.war

When the Management Center starts, you should see the logs below:

Started AppDynamics Java Agent Successfully.
Hazelcast Management Center starting on port 8080 at path : /

21. Management Center Configuration Tool

The Management Center Configuration Tool (MC-Conf) is a command line tool that allows you to update certain parts of the Management Center configuration by using its built-in tasks. You can use the mc-conf.sh or mc-conf.bat script to run the MC-Conf tool.

You must run the MC-Conf tool on the same machine where the Management Center web application is deployed.

21.1. Built-In Help

In order to see all available commands, run the MC-Conf script with no arguments as shown below.

./mc-conf.sh

As the result, you should see an output similar to below.

Hazelcast Management Center Configuration Tool 4.0
Usage: mc-conf [-hV] COMMAND TASK
Command line tool for interacting with Hazelcast Management Center
configuration.

Global options are:
  -h, --help      Show this help message and exit.
  -V, --version   Print version information and exit.
Commands:
  cluster           Manage Cluster Connection Configs
  user              Manage Default Security Provider Users
  ldap              Manage LDAP Security Provider
  active-directory  Manage Active Directory Security Provider
  jaas              Manage JAAS Security Provider
  security          General Security Provider management

When you choose a specific subcommand from the list above, you can see all tasks available for it. See the following example:

$ ./mc-conf.sh user
Usage: mc-conf user [-hV] TASK
Manage Default Security Provider Users
  -h, --help      Show this help message and exit.
  -V, --version   Print version information and exit.
Commands:
  create           Create a new user record in the default security provider.
                   *Important notice* Make sure that Management Center web
                     application is stopped (offline) before starting this
                     task.

  update-password  Change password for the given user record in the default
                     security provider.
                   *Important notice* Make sure that Management Center web
                     application is stopped (offline) before starting this
                     task.

You can also get help for any task by using the -h (or --help) command line option. See the following example:

$ ./mc-conf.sh user create -h
Usage: mc-conf user create [-hvV] [-p[=<password>]] [-H=<homedir>]
                           -n=<username> -r=<role>
Create a new user record in the default security provider.
*Important notice* Make sure that Management Center web application is stopped
(offline) before starting this task.

  -H, --home=<homedir>   Optional path to Management Center home directory. By
                           default ~/hazelcast-mc/ is used.
  -n, --username=<username>
                         Username for the user record.
  -p, --password[=<password>]
                         Password for the user record. Provide value directly
                           or use without value to enter securely with
                           interactive prompt.
  -r, --role=<role>      Roles for the user record. Valid values: readonly,
                           readwrite, metricsonly, admin.
  -h, --help             Show this help message and exit.
  -V, --version          Print version information and exit.
  -v, --verbose          Show logs from Management Center and full stack trace
                           of errors.

21.2. Configuring Cluster Connection

The cluster add task adds a new connection configuration for a cluster. Note that you must stop the Management Center web application before running this task.

You can use this task for various scripting purposes, and automatically configuring Management Center, without the need for a manual cluster connection configuration through UI.

If you have used a non-default Management Center home directory location, you must provide the path to the home directory with the -H (or --home) option.

21.3. Creating Users

The user create task creates a new user in the default security provider. Note that you must stop the Management Center web application before running this task.

You can use this task for various scripting purposes. See the Hazelcast Docker Code Samples repository for an example of Docker image for the Management Center container with a built-in user account.

If you have used a non-default Management Center home directory location, you must provide the path to the home directory with the -H (or --home) option.

21.4. Changing User Password

The user update-password task resets the password of a specified user in the default security provider. Note that you must stop the Management Center web application before running this task.

You can use this task as a recovery mechanism for the Management Center’s administrator user account.

If you have used a non-default Management Center home directory location, you must provide the path to the home directory with the -H (or --home) option.

21.5. Configuring LDAP Security Provider

The ldap configure task configures the LDAP security provider. Note that you must stop the Management Center web application, before running this task.

You can use this task for various scripting purposes and automatically configuring Management Center without the need for a manual security provider configuration through UI.

You can encrypt the LDAP password before saving with this task. See the Variable Replacers section for more information.

As with the UI based LDAP configuration, you can also use keystore for secure password storage, by using the optional --key-store-* options, as shown in the examples below.

If you want to use the built-in Management Center managed keystore, you can add the following options: --ks-create --key-store=<hazelcast-mc directory>/mc.jceks --key-store-password=<password>. This creates a keystore in the default Management Center directory, and saves the LDAP password in it. If you want to customize the keystore name or Management Center directory when starting Management Center, you need to reflect that with the --key-store=<path> option.

If you want to use the existing externally managed keystore, you can use the following options: --key-store=<keystore path> --key-store-password=<password> [--key-store-type=<type> --key-store-provider=<provider>]. Note that if the keystore with such path doesn’t exist, task fails.

You still need to properly configure Management Center to use keystore. See LDAP Authentication section for details on using the built-in and existing keystores.
If you have used a non-default Management Center home directory location, you must provide the path to the home directory with the -H (or --home) option.

21.6. Updating LDAP Password

The ldap update-password task updates the encrypted LDAP password stored in the keystore. It expects information about the keystore such as its location and password and the new LDAP password that you want to use. See the LDAP Authentication section for more information on the encrypted LDAP passwords. After updating the LDAP password, you need to click on the Reload Security Config button on the login page.

21.7. Configuring Active Directory Security Provider

The active-directory configure task configures the Active Directory security provider. Note that you must stop the Management Center web application before running this task.

You can use this task for various scripting purposes, and automatically configuring Management Center, without the need for a manual security provider configuration through UI.

If you have used a non-default Management Center home directory location, you must provide the path to the home directory with the -H (or --home) option.

21.8. Configuring JAAS Security Provider

The jaas configure task configures the JAAS security provider. Note that you must stop the Management Center web application before running this task.

You can use this task for various scripting purposes, and automatically configuring Management Center, without the need for a manual security provider configuration through UI.

If you have used a non-default Management Center home directory location, you must provide the path to the home directory with the -H (or --home) option.

21.9. Resetting Security Provider

The security reset task resets current security provider used in the Management Center. For the default security provider it also deletes all built-in user accounts. Note that you must stop the Management Center web application before running this task.

You can use this task as a recovery mechanism for the Management Center deployment in case if a non-default security provider is configured. In case of the default security provider, you can also use the user create or user update-password task as the recovery mechanism.

If you have used a non-default Management Center home directory location, you must provide the path to the home directory with the -H (or --home) option.

21.10. Advanced Features

MC-Conf supports interactive options for secure processing of passwords. To use it, you need to use the password option without providing a value, i.e., instead of --password=<password> use --password. When you use this option without providing a value, you will get a prompt to enter a value on the console. An example of the interactive option usage is shown below.

$ ./mc-conf.sh user update-password --username=admin --password
Enter value for --password (Password for the user record. Provide value directly,
or use without value to enter securely with interactive prompt.): ********
Successfully changed password for user 'admin'.

As you see in the above example, the password input is not echoed to the console since it is provided with the secure interactive mode.

Another advanced feature of MC-Conf is the support for argument files. When an argument beginning with the character @ is encountered, it is treated as a path leading to a text file. The contents of that file are automatically expanded into the current task. An example of the argument file usage is shown below.

$ ./mc-conf.sh user update-password @arg-file.txt
Successfully changed password for user 'admin'.
$ cat arg-file.txt
--username=admin --password=mnb3c4s0

22. Phone Home

Hazelcast uses phone home data to learn about usage of Hazelcast Management Center.

Hazelcast Management Center instances initially call our phone home server 30 minutes after they are started and once every 24 hours thereafter.

What is sent in?

The following information is sent in a phone home:

  • Hazelcast Management Center version

  • Authentication provider used (Default, LDAP, ActiveDirectory, JAAS)

  • Whether clustered REST is enabled or not

  • Whether clustered JMX is enabled or not

  • Whether TLS is enabled or not

    • If TLS is enabled, whether mutual authentication is enabled or not

  • Whether Management Center is deployed on an application server or used in standalone mode

    • If not in standalone mode, type of the application server

  • Number of users (if the default security provider is used)

  • Number of clusters

  • Management Center uptime

  • Minimum and maximum cluster sizes

  • Minimum and maximum cluster versions

  • Total number of members

  • Size of the Hazelcast Management Center home directory

  • Hash value of Hazelcast Management Center license key

  • Environment Information:

    • Name of operating system

    • Version of installed Java

For each user login, we store the following information and send it in a phone home:

  • Browser (Chrome, Firefox, IE etc.)

  • Browser major version

  • Operating system

  • Operating system version

  • Screen height and width

  • Window height and width

Disabling Phone Homes

Set the hazelcast.mc.phone.home.enabled system property to false on the Java command line.

Phone Home URL

http://phonehome.hazelcast.com/pingMc

23. Management Center Documentation

To see the Management Center documentation (this Reference Manual), click on the Documentation button located at the toolbar. This Management Center manual appears as a tab.

24. Suggested Heap Size

Table 1. For 2 Cluster Members
Management Center Heap Size # of Maps # of Queues # of Topics

256m 

3k

1k

1k

1024m

10k

1k

1k

Table 2. For 10 Members
Management Center Heap Size # of Maps # of Queues # of Topics

256m 

50

30

30

1024m

2k

1k

1k

Table 3. For 20 Members
Management Center Heap Size # of Maps # of Queues # of Topics

256m [1]

N/A

N/A

N/A

1024m

1k

1k

1k

25. Troubleshooting

25.1. Horizontal Scrollbar in Tables when Using macOS

MacOS automatically shows a horizontal scrollbar in the status tables of Management Center, when you scroll through a table content. See below for an example view:

MacOS Scroll Bar

The scrollbar should hide shortly after you stop scrolling. This is the default behavior and it shouldn’t cause any inconvenience. However, if you still want to change it, address "Show scroll bars" section of the "Change General preferences on Mac" guide.

Appendix A: Migration Guides

This appendix provides information on compatibility related changes for Hazelcast Management Center releases.

A.1. Hazelcast Management Center 3.12.x

  • Default home directory location has been changed from <user-home>/hazelcast-mancenter-<version> to <user-home>/hazelcast-mc.

  • Parameter to change home directory location has been changed from hazelcast.mancenter.home to hazelcast.mc.home.

  • The UpdateLdapPassword utility (available via updateLdapPassword.sh or updateLdapPassword.bat scripts) has been merged into the MC Conf tool (available via the mc-conf.sh or mc-conf.bat scripts).

A.2. Hazelcast Management Center 3.10.x

  • Hazelcast Management Center’s default URL has been changed from localhost:8080/mancenter to localhost:8080/hazelcast-mancenter.

  • Default home directory location has been changed from <user-home>/mancenter-<version> to <user-home>/hazelcast-mancenter-<version>.

  • Name of the WAR file has been changed from mancenter-{version}.war to hazelcast-mancenter-{version}.war.

A.3. Hazelcast Management Center 3.8.x

Starting with Management Center 3.8.4, you can use the following system properties for Clustered JMX via Management Center:

  • -Dhazelcast.mc.jmx.rmi.port=9001

  • -Dhazelcast.mc.jmx.host=localhost

Appendix B: User Interface Logs

Using the "UI Logs" page of the "Settings" menu, you can see the log entries related to the Management Center user interface. This page looks like the following:

User Interface Logs

This is basically useful to make the troubleshooting easier related to the issues in the user interface. You can see the timestamp and type of each log entry.

You can reorder the entries by their timestamps any types, and also filter them by giving a keyword to the Data field such as "auth", "map" and "session".

You can pause the log flow using the Pause button, copy the log entries to the clipboard using the Copy button (so that the entries can be examined in detail) and resume the flow using the Resume button on the page.

You may only need to share the information in these logs if requested by the Hazelcast’s support team.

1. With 256m heap, Management Center is unable to collect statistics.