Hazelcast Management Center Reference Manual

Here are the steps.

When starting Management Center from the command line, a license can be provided using the system property hazelcast.mc.license. For example by using the command line parameter:

When this option is used the license provided will take precedence over any license set and stored previously using the user interface. Previously stored licenses are not affected and will be used again when the Management Center is started without the hazelcast.mc.license property. This also means no new license can be stored when the property is used.

Starting with 3.10, the disk space used by Management Center is constraint to avoid exceeding available disk space. When the set limit is exceeded Management Center deals with this in two ways:

Usually, either of the above automatically resolves the situation and makes room for new data. Depending on the disk usage configuration and the kind of data that contributes to exceeding the limit it can occur that the limit continues to be exceeded. In this case Management Center will not store new alerts or metrics data. Other data (like configurations and account information) is still stored as they hardly cause larger data volumes.

An active blockage is reported in the UI as an error notification, as shown below:

However, storage operations will not explicitly fail or report errors since this would constantly cause interruptions and error logging - both in the UI and logs.

One way to resolve a blockage is deleting the data manually, e.g., deleting a filter that caused many alerts in the alerts view. Another way is to restart Management Center with a higher limit or in purge mode (if not used before).

You can use the following system properties to configure Management Center’s disk usage control:

It is important to understand that the limit given is a soft limit, a high water mark. Management Center will act if it is exceeded but it might be exceeded by a margin between two measurements. Do not set it to the absolute maximum disk space available. A smaller interval increases accuracy but also performance overhead.

In case of a misconfiguration of any of the three properties Management Center will log the problem and abort startup immediately.

When you start Management Center from the command line, it will serve the pages unencrypted by using "http", by default. To enable TLS/SSL, use the following command line parameters when starting the Management Center:

You can leave trust store and trust store password values empty to use the system JVM’s own trust store.

Following is an example on how to start Management Center with TLS/SSL enabled from the command line:

You can access Management Center from the following HTTPS URL on port 8443: https://localhost:8443/hazelcast-mancenter

To override the HTTPS port, you can give it as the second argument when starting Management Center. For example:

This will start Management Center on HTTP port 80 and HTTPS port 443 with context path /hazelcast-mancenter. Note that accessing port 80 with an http:// prefix will redirect the users to an https:// URL on port 443. It means that the users will use HTTPS regardless of the version of the URL they use.

For increased security, you can disable the HTTP port completely and use the HTTPS port only. Use the following command line argument:

Mutual authentication allows cluster members to have their keyStores and Management Center to have their trustStores so that Management Center can know which members it can trust. To enable mutual authentication, you need to use the following command line parameters when starting the Management Center:

And at member side, you need to set the following JVM arguments when starting the member:

Please see the below example snippet to see the full command to start Management Center:

And the full command to start the cluster member:

The parameter -Dhazelcast.mc.tls.mutualAuthentication has two options:

If you have started Management Center from the command line by using the WAR file, by default, sessions that are inactive for 30 minutes are invalidated. To change this, you can use the -Dhazelcast.mc.session.timeout.seconds command line parameter.

For example, the following command starts Management Center with a session timeout period of 1 minute:

If you have deployed Management Center on an application server/container, you can configure the default session timeout period of the application server/container to change the session timeout period for Management Center. If your server/container allows application specific configuration, you can use it to configure the session timeout period for Management Center.

Normally, a user account on Management Center can’t be used from multiple locations at the same time. If you want to allow others to log in, when there’s already someone logged in with the same username, you can start Management Center with the -Dhazelcast.mc.allowMultipleLogin=true command line parameter.

In order to prevent password guessing attacks, logging in is disabled temporarily after a number of failed login attempts. When not configured explicitly, default values are used, i.e., logging in is disabled for 5 seconds when a username is failed to log in consecutively 3 times. During this 5 seconds of period, logging in will not be allowed even when the correct credentials are used. After 5 seconds, the user will be able to log in using the correct credentials.

Assuming the configuration with the default values, if the failed attempts continue (consecutively 3 times) after the period of disabled login passes, this time the disable period will be multiplied by 10 and logging in will be disabled for 50 seconds; the whole process repeats itself until the user logs in successfully. By default, there’s no upper limit to the disable period, but can be configured by using the -Dhazelcast.mc.maxDisableLoginPeriod parameter.

Here is a scenario, in the given order, with the default values:

You can configure the number of failed login attempts, initial and maximum duration of the disabled login and the multiplier using the following command line parameters:

If you haven’t allowed multiple simultaneous login attempts explicitly, the first user to login with a username stays logged in until that username explicitly logs out or its session expires. In the meantime, no one else can login with the same username. If you want to force logout for the first user and let the newcomer login, you need to start Management Center with the -Dhazelcast.mc.forceLogoutOnMultipleLogin=true command line parameter.

In order to prevent certain words from being included in the user passwords, you can start the Management Center with -Dhazelcast.mc.security.dictionary.path command line parameter which points to a text file that contains a word on each line. As a result, the user passwords will not contain any dictionary words, making them harder to guess.

The words in the dictionary need to be at least 3 characters long in order to be used for checking the passwords. The shorter words will be ignored to prevent them from blocking the usage of many password combinations. You can configure the minimum length of words by starting the Management Center with -Dhazelcast.mc.security.dictionary.minWordLength command line parameter and setting it to a number.

An example to start the Management Center using the aforementioned parameters is shown below:

You can also start the Management Center with an extra classpath entry (for example, when using JAAS authentication) by using the following command:

On Windows, the command becomes as follows (semicolon instead of colon):

Optionally, you can use the scripts start.bat or start.sh to start the Management Center.

Or, instead of starting at the command line, you can deploy it to your application server (Tomcat, Jetty, etc.).

If you have deployed hazelcast-mancenter-3.10.4.war in your already-SSL-enabled web container, configure hazelcast.xml as follows.

If you are using an untrusted certificate for your container, which you created yourself, you need to add that certificate to your JVM first. Download the certificate from the browser, after this you can add it to JVM as follows.

After you perform the above steps, make sure that http://localhost:8080/hazelcast-mancenter is up.

Configure your Hazelcast members by adding the URL of your web application to your hazelcast.xml. Hazelcast members will send their states to this URL.

You can configure it programmatically as follows.

If you enabled TLS/SSL on Management Center, then you will need to configure the members with the relevant keystore & trustore. In that case you expand the above configuration as follows.

In the example above, Hazelcast’s default SSL context factory (BasicSSLContextFactory) is used; you can also provide your own implementation of this factory.

Here are the descriptions for the properties:

See the programmatic configuration example below:

Now you can start your Hazelcast cluster, browse to http://localhost:8080/hazelcast-mancenter or https://localhost:sslPortNumber/hazelcast-mancenter (depending on installation) and setup your administrator account explained in the next section.

If a Hazelcast cluster is configured to use TLS for communication between its members using a self-signed certificate, Management Center will not be able to perform some of the operations that use the cluster’s HTTP endpoints (such as shutting down a member or getting the thread dump of a member). This is so because self-signed certificates are not trusted by default by the JVM. For these operations to work, you need to configure a truststore containing the public key of the self-signed certificate when starting the JVM of Management Center using the following command line parameters:

By default, JVM also checks for the validity of the hostname of the certificate. If this test fails, you will see a line similar to the following in the Management Center logs:

If you want to disable this check, you will need to start Management Center with the following command line parameter:

You can set a frequency (in seconds) for which Management Center will take information from the Hazelcast cluster, using the element update-interval as shown below. update-interval is optional and its default value is 3 seconds.

Management Center uses Logback for its logging. By default, it uses the following configuration:

To change the logging configuration, you can create a custom Logback configuration file and start Management Center with the -Dlogback.configurationFile option pointing to your configuration file.

For example, you can create a file named logback-custom.xml with the following content and set logging level to DEBUG. To use this file as the logging configuration, you need to start Management Center with -Dlogback.configurationFile=/path/to/your/logback-custom.xml command line parameter:

This example EncryptionReplacer replaces encrypted variables by its plain form. The secret key for encryption/decryption is generated from a password which can be a value in a file and/or environment specific values, such as MAC address and actual user data.

Its full class name is com.hazelcast.webmonitor.configreplacer.EncryptionReplacer and the replacer prefix is ENC. Here are the properties used to configure this example replacer:

As a usage example, let’s create a password file and generate the encrypted strings out of this file.

1 - Create the password file: echo '/Za-uG3dDfpd,5.-' > /opt/master-password

2 - Define the encrypted variables:

3 - Configure the replacer and provide the encrypted variables as command line arguments while starting Management Center:

The PropertyReplacer replaces variables by properties with the given name. Usually the system properties are used, e.g., ${user.name}.

Its full class name is com.hazelcast.webmonitor.configreplacer.PropertyReplacer and the replacer prefix is empty string ("").

You can also provide your own replacer implementations. All replacers have to implement the three methods that have the same signatures as the methods of the following interface:

You can use your existing Active Directory server for authentication/authorization on Management Center. In the "Configure Security" page, select Active Directory from the "Security Provider" combo, and the following form page appears:

Provide the details in this form for your Active Directory server:

Once configured, Active Directory settings are saved in a file named ldap.properties under the hazelcast-mancenter3.10.4 folder mentioned in the previous section. If you want to update your settings afterwards, you need to update ldap.properties file and click "Reload Security Config" button on the login page.

You can use your own javax.security.auth.spi.LoginModule implementation for authentication/authorization on Management Center. In the "Configure Security" page, select JAAS from the "Security Provider" combo box, and the following page appears:

Provide the details in this form for your JAAS LoginModule implementation:

Following is an example implementation. Note that we return two java.security.Principal instances; one of them is the username and the other one is a group name, which you will use when configuring JAAS security as described above.

You can use your existing LDAP server for authentication/authorization on Management Center. In the "Configure Security" page, select LDAP from the "Security Provider" combo box, and the following form page appears:

Provide the details in this form for your LDAP server:

Once configured, LDAP settings are saved in a file named ldap.properties under the hazelcast-mancenter3.10.4 folder mentioned in the previous section. If you want to update your settings afterwards, you need to update ldap.properties file and click "Reload Security Config" button on the login page.

The toolbar has the following elements:

The user can select any cluster and once selected, the page immediately loads with the selected cluster’s information.

The Home Page includes a menu on the left which lists the distributed data structures in the cluster, cluster members and clients connected to the cluster (numbers in square brackets show the instance count for each entity), as shown below. You can also see an overview state of your cluster, create alerts, execute codes and perform user/license operations using this menu:

Below is the list of menu items with links to their explanations.

This part of the page provides information related to memory usages for each member, as shown below.

The first column lists the members with their IPs and ports. The next columns show the used and free memories out of the total memory reserved for Hazelcast usage, in real-time. The Max. Heap column lists the maximum memory capacity of each member and the Heap Usage Percentage column lists the percentage value of used memory out of the maximum memory. The Used Heap column shows the memory usage of members graphically. When you move the mouse cursor on a desired graph, you can see the memory usage at the time where the cursor is placed. Graphs under this column shows the memory usages approximately for the last 2 minutes.

This part of the page graphically provides the cluster wise breakdown of heap memory, as shown below. The blue area is the heap memory used by maps (including all owned/backup entires and any near cache usage). The dark yellow area is the heap memory used by both non-Hazelcast entities and all Hazelcast entities except the map (i.e. the heap memory used by all entities subtracted by the heap memory used by map). The green area is the free heap memory out of the whole cluster’s total committed heap memory.

In the above example, you can see 26.18% of the total heap memory is used by Hazelcast maps, 36.02% is used by both non-Hazelcast entities and all Hazelcast entities except the map and 37.80% of the total heap memory is free.

This part provides the percentage values of the memories used by each map, out of the total cluster memory reserved for all Hazelcast maps.

In the above example, you can see 62.50% of the total map memory is used by Map A and 37.50% is used by Map B.

This part shows the current cluster state and the cluster’s health. For more information on cluster states, see Cluster State. Cluster health shows how many migrations are taking place currently.

This pie chart shows what percentage of partitions each cluster member has, as shown below.

You can see each member’s partition percentages by placing the mouse cursor on the chart. In the above example, you can see the member "127.0.0.1:5702" has 33.21% of the total partition count (which is 271 by default and configurable, please see the hazelcast.partition.count property explained in the System Properties section.

This part of the page provides load and utilization information for the CPUs for each cluster member, as shown below.

The first column lists the members with their IPs and ports. The next columns list the system load averages on each member for the last 1, 5 and 15 minutes. These average values are calculated as the sum of the count of runnable entities running on and queued to the available CPUs averaged over the last 1, 5 and 15 minutes. This calculation is operating system specific, typically a damped time-dependent average. If system load average is not available, these columns show negative values.

The last column (Utilization(%)) graphically shows the recent load on the CPUs. When you move the mouse cursor on a chart, you can see the CPU load at the time where the cursor is placed. Charts under this column shows the CPU loads approximately for the last 2 minutes. If recent CPU load is not available, you will see a negative value.

Use the Map Browser tool to retrieve properties of the entries stored in the selected map. To open the Map Browser tool, click on the Map Browser button, located at the top right of the window. Once opened, the tool appears as a dialog, as shown below.

Once the key and the key’s type are specified and the Browse button is clicked, the key’s properties along with its value are listed.

Use the Map Config tool to set the selected map’s attributes, such as the backup count, TTL, and eviction policy. To open the Map Config tool, click on the Map Config button, located at the top right of the window. Once opened, the tool appears as a dialog, as shown below.

You can change any attribute and click the Update button to save your changes.

Besides the Map Browser and Map Config tools, the map monitoring page has monitoring options that are explained below. All of these options perform real-time monitoring.

On top of the page, small charts monitor the size, throughput, memory usage, backup size, etc. of the selected map in real-time. The X-axis of all the charts show the current system time. You can select other small monitoring charts using the button at the top right of each chart. When you click the button, the monitoring options are listed, as shown below.

When you click on a desired monitoring, the chart is loaded with the selected option. To open a chart as a separate dialog, click on the button placed at the top right of each chart. The monitoring charts below are available:

Under these charts are Map Memory and Map Throughput data tables. The Map Memory data table provides memory metrics distributed over members, as shown below.

From left to right, this table lists the IP address and port, entry counts, memory used by entries, backup entry counts, memory used by backup entries, events, hits, locks and dirty entries (in the cases where MapStore is enabled, these are the entries that are put to/removed from the map but not written to/removed from a database yet) of each entry in the map. You can navigate through the pages using the buttons at the bottom right of the table (First, Previous, Next, Last). You can ascend or descend the order of the listings by clicking on the column headings.

Map Throughput data table provides information about the operations (get, put, remove) performed on each member in the map, as shown below.

From left to right, this table lists:

You can select the period in the combo box placed at the top right corner of the window, for which the table data will be shown. Available values are Since Beginning, Last Minute, Last 10 Minutes and Last 1 Hour.

You can navigate through the pages using the buttons placed at the bottom right of the table (First, Previous, Next, Last). To ascend or descent the order of the listings, click on the column headings.

A WAN publisher can be in one of the following states:

You can change a WAN publisher’s state by clicking the Change State dropdown button on top right hand corner of the WAN Replication Operations Table.

You can initiate a synchronization operation on an IMap for a specific target cluster. Synchronization operation sends all the data of an IMap to a target cluster to align the state of target IMap with source IMap. Synchronization is useful if two remote clusters lost their synchronization due to WAN queue overflow or in restart scenarios.

To initiate WAN Sync, open the WAN Replication menu item on the left and navigate to the Sync tab.

Click Start WAN Sync button to open the dialog, enter the target details for the sync operation and click Sync to start the operation.

You can also use the "All Maps" option in the above dialog if you want to synchronize all the maps in source and target cluster.

You can see the progress of the operation once you initiate it.

You can add a WAN replication configuration dynamically to a cluster. It is useful for having one-off WAN sync operations. The added configuration has two caveats:

See the Hazelcast IMDG Reference Manual section on WAN Replication for details about the fields and their possible values.

After clicking Add Configuration button, the new WAN replication configuration is added to the cluster. You can see the new configuration when you try to initiate a WAN sync operation as described in the previous section.

Select Member Alerts check box to create filters for some or all members in the cluster. Once selected, the next screen asks for which members the alert will be created. Select the desired members and click on the Next button. On the next page (shown below), specify the criteria.

You can create alerts when:

When two or more criteria is specified they will be bound with the logical operator AND.

On the next page, give a name for the filter. Then, select whether notification e-mails will be sent to the Management Center Admins using the Send Email Alert checkbox. Then, provide a time interval (in seconds) for which the e-mails with the same notification content will be sent using the Email Interval (secs) field. Finally, select whether the alert data will be written to the disk (if checked, you can see the alert log at the folder /users/<your user>/hazelcast-mancenter3.10.4).

Click on the Save button; your filter will be saved and put into the Filters part of the page. To edit the filter, click on the icon. To delete it, click on the icon.

Select the Data Type Alerts check box to create filters for data structures. The next screen asks for which data structure (maps, queues, multimaps, executors) the alert will be created. Once a structure is selected, the next screen immediately loads and you then select the data structure instances (i.e. if you selected Maps, it will list all the maps defined in the cluster, you can select one map or more). Select as desired, click on the Next button, and select the members on which the selected data structure instances will run.

The next screen, as shown below, is the one where you specify the criteria for the selected data structure.

As the screen shown above shows, you will select an item from the left combo box, select the operator in the middle one, specify a value in the input field, and click on the Add button. You can create more than one criteria in this page; those will be bound by the logical operator AND.

After you specify the criteria, click the Next button. On the next page, give a name for the filter. Then, select whether notification e-mails will be sent to the Management Center Admins using the Send Email Alert checkbox. Then, provide a time interval (in seconds) for which the e-mails with the same notification content will be sent using the Email Interval (secs) field. Finally, select whether the alert data will be written to the disk (if checked, you can see the alert log at the folder /users/<your user>/hazelcast-mancenter3.10.4).

Click on the Save button; your filter will be saved and put into the Filters part of the page. To edit the filter, click on the icon. To delete it, click on the icon.

The admin user can see and change the cluster state and shut down the cluster using the operations listed in this screen as shown below.

Cluster States:

Changing Cluster State

Shutting Down the Cluster

If an exception occurs during the state change or shutdown operation on the cluster, this exception message will be shown on the screen as a notification.

To update the Management Center license, you can open the Manage License tab and click Update License button and enter the new license code.

Alternatively a license can be provided using the system property hazelcast.mc.license (see Starting with a License for details).

If the Hazelcast cluster is configured to use a socket interceptor, you need to configure a socket interceptor for Management Center as well. You can click on the Configure Socket Interceptor button, enter the name of your socket interceptor class and the configuration parameters. Click the Configure button to save your configuration and enable socket interceptor. This class needs to be on your classpath when you’re starting Management Center. The configuration parameters you provide will be used to invoke the init method of your socket interceptor implementation if it has such a method.

Following is a sample socket interceptor class implementation:

A socket interceptor implementation needs to satisfy the following two conditions:

Hazelcast cluster members need to be configured with Management Center’s URL before they’re started. If Management Center’s URL is changed for some reason, you can use this page to make Hazelcast members send their statistics to the new Management Center URL. However, this has the following caveats:

To change the URL, enter the Cluster Name and Password, provide the IP address and port for one of the members, and specify the new Management Center URL in the Server URL field. If the cluster members are configured to use TLS/SSL for communicating between themselves, check the SSL box. Clicking the Set URL button will update the Management Center URL.

To add a user to the system, specify the username, e-mail and password in the Add/Edit User part of the page. If the user to be added will have administrator privileges, select isAdmin checkbox. Permissions field has the following checkboxes:

After you enter/select all fields, click Save button to create the user. You will see the newly created user’s username on the left side, in the Users part of the page.

To edit or delete a user, select a username listed in the Users. Selected user information appears on the right side of the page. To update the user information, change the fields as desired and click the Save button. You can also change a user’s password or delete the user account. To change the user’s password, click the Change Password button. To delete the user from the system, click the Delete button. Note that changing the password of a user and deleting the user account both require you to enter your own password.

The admin user can upgrade the cluster version once all members of the cluster have been upgraded to the intended codebase version as described in the Rolling Upgrade Procedure section of the Hazelcast IMDG Reference Manual.

Open the Rolling Upgrade tab to perform a rolling upgrade and change the cluster’s version.

Enter the group name/password of the cluster and the version you want to upgrade the cluster to, and click on the Change Version button.

Once the operation succeeds, you will see the following notification:

Force Start the Cluster

Restart process cannot be completed if a member crashes permanently and cannot recover from the failure since it cannot start or it fails to load its own data. In that case, you can force the cluster to clean its persisted data and make a fresh start. This process is called force start.

Click the Force Start button. A pop-up will appear and stay on screen until the operation is triggered.

If an exception occurs, this exception message will be showed on the screen as a notification.

To enable Clustered REST on your Management Center, pass the following system property at startup. This property is disabled by default.

The entry point for Clustered REST API is /rest/.

This resource does not have any attributes.

This resource returns a list of clusters that are connected to the Management Center.

This resource returns information related to the provided cluster name.

This resource returns a list of members belonging to the provided clusters.

This resource returns information related to the provided member.

This resource returns a list of client endpoints belonging to the provided cluster. Please consider using the newly added Client Statistics Resource as it contains more detailed information about clients.

This resource returns a list of maps belonging to the provided cluster.

This resource returns a list of multimaps belonging to the provided cluster.

This resource returns a list of replicated maps belonging to the provided cluster.

This resource returns a list of queues belonging to the provided cluster.

This resource returns a list of topics belonging to the provided cluster.

This resource returns a list of executors belonging to the provided cluster.

This resource returns a list of clients belonging to the provided cluster.

In order to configure Clustered JMX, use the following command line parameters for your Management Center deployment.

Starting with Hazelcast Management Center 3.8.4, you can also use the following parameters:

With embedded Jetty, you do not need to deploy your Management Center application to any container or application server.

You can start Management Center application with Clustered JMX enabled as shown below.

Once Management Center starts, you should see a log similar to below.

You should be able to connect to Clustered JMX interface from the address localhost:9999.

You can use jconsole or any other JMX client to monitor your Hazelcast Cluster. As a sample, below is the jconsole screenshot of the Clustered JMX hierarchy.

The management beans are exposed with the following object name format.

Object name starts with ManagementCenter prefix. Then it has the cluster name in brackets followed by a colon. After that, type,name and member attributes follows, each separated with a comma.

A sample bean is shown below.

Here is the list of attributes that are exposed from the Clustered JMX interface.

Use the Clustered JMX interface to integrate Hazelcast Management Center with New Relic. To perform this integration, attach New Relic Java agent and provide an extension file that describes which metrics will be sent to New Relic.

Please see Custom JMX instrumentation by YAML on the New Relic webpage.

Below is an example Map monitoring .yml file for New Relic.

Put the .yml file in the extensions folder in your New Relic installation. If an extensions folder does not exist there, create one.

After you set your extension, attach the New Relic Java agent and start Management Center as shown below.

If your logging level is set as FINER, you should see the log listing in the file newrelic_agent.log, which is located in the logs folder in your New Relic installation. Below is an example log listing.

Then you can navigate to your New Relic account and create Custom Dashboards. Please see Creating custom dashboards.

While you are creating the dashboard, you should see the metrics that you are sending to New Relic from Management Center in the Metrics section under the JMX folder.

Use the Clustered JMX interface to integrate Hazelcast Management Center with AppDynamics. To perform this integration, attach AppDynamics Java agent to the Management Center.

For agent installation, refer to Install the App Agent for Java page.

For monitoring on AppDynamics, refer to Using AppDynamics for JMX Monitoring page.

After installing AppDynamics agent, you can start Management Center as shown below.

When Management Center starts, you should see the logs below.