Partition Group Configuration - Hazelcast Reference Manual

Hazelcast distributes key objects into partitions using the consistent hashing algorithm. Multiple replicas are created for each partition and those partition replicas are distributed among Hazelcast members. An entry is stored in the members that own replicas of the partition to which the entry's key is assigned. The total partition count is 271 by default; you can change it with the configuration property hazelcast.partition.count. Please see the System Properties section.

Hazelcast member that owns the primary replica of a partition is called as partition owner. Other replicas are called backups. Based on the configuration, a key object can be kept in multiple replicas of a partition. A member can hold at most one replica of a partition (ownership or backup).

By default, Hazelcast distributes partition replicas randomly and equally among the cluster members, assuming all members in the cluster are identical. But what if some members share the same JVM or physical machine or chassis and you want backups of these members to be assigned to members in another machine or chassis? What if processing or memory capacities of some members are different and you do not want an equal number of partitions to be assigned to all members?

To deal with such scenarios, you can group members in the same JVM (or physical machine) or members located in the same chassis. Or you can group members to create identical capacity. We call these groups partition groups. Partitions are assigned to those partition groups instead of individual members. Backup replicas of a partition which is owned by a partition group are located in other partition groups.

Grouping Types

When you enable partition grouping, Hazelcast presents the following choices for you to configure partition groups.

1. HOST_AWARE:

You can group members automatically using the IP addresses of members, so members sharing the same network interface will be grouped together. All members on the same host (IP address or domain name) will be a single partition group. This helps to avoid data loss when a physical server crashes, because multiple replicas of the same partition are not stored on the same host. But if there are multiple network interfaces or domain names per physical machine, that will make this assumption invalid.

Following are declarative and programmatic configuration snippets that show how to enable HOST_AWARE grouping.

<partition-group enabled="true" group-type="HOST_AWARE" />

Config config = ...;
PartitionGroupConfig partitionGroupConfig = config.getPartitionGroupConfig();
partitionGroupConfig.setEnabled( true )
    .setGroupType( MemberGroupType.HOST_AWARE );

2. CUSTOM:

You can do custom grouping using Hazelcast's interface matching configuration. This way, you can add different and multiple interfaces to a group. You can also use wildcards in the interface addresses. For example, the users can create rack-aware or data warehouse partition groups using custom partition grouping.

Following are declarative and programmatic configuration examples that show how to enable and use CUSTOM grouping.

<partition-group enabled="true" group-type="CUSTOM">
<member-group>
  <interface>10.10.0.*</interface>
  <interface>10.10.3.*</interface>
  <interface>10.10.5.*</interface>
</member-group>
<member-group>
  <interface>10.10.10.10-100</interface>
  <interface>10.10.1.*</interface>
  <interface>10.10.2.*</interface>
</member-group
</partition-group>

Config config = ...;
PartitionGroupConfig partitionGroupConfig = config.getPartitionGroupConfig();
partitionGroupConfig.setEnabled( true )
    .setGroupType( MemberGroupType.CUSTOM );

MemberGroupConfig memberGroupConfig = new MemberGroupConfig();
memberGroupConfig.addInterface( "10.10.0.*" )
.addInterface( "10.10.3.*" ).addInterface("10.10.5.*" );

MemberGroupConfig memberGroupConfig2 = new MemberGroupConfig();
memberGroupConfig2.addInterface( "10.10.10.10-100" )
.addInterface( "10.10.1.*").addInterface( "10.10.2.*" );

partitionGroupConfig.addMemberGroupConfig( memberGroupConfig );
partitionGroupConfig.addMemberGroupConfig( memberGroupConfig2 );

3. PER_MEMBER:

You can give every member its own group. Each member is a group of its own and primary and backup partitions are distributed randomly (not on the same physical member). This gives the least amount of protection and is the default configuration for a Hazelcast cluster. This grouping type provides good redundancy when Hazelcast members are on separate hosts. However, if multiple instances run on the same host, this type is not a good option.

Following are declarative and programmatic configuration snippets that show how to enable PER_MEMBER grouping.

<partition-group enabled="true" group-type="PER_MEMBER" />

Config config = ...;
PartitionGroupConfig partitionGroupConfig = config.getPartitionGroupConfig();
partitionGroupConfig.setEnabled( true )
    .setGroupType( MemberGroupType.PER_MEMBER );

4. ZONE_AWARE:

You can use ZONE_AWARE configuration with Hazelcast jclouds or Hazelcast Azure Discovery Service plugins.

As discovery services, these plugins put zone, rack, and host information to the Hazelcast member attributes map during the discovery process. Hazelcast creates the partition groups with respect to member attributes map entries that include zone, rack, and host information, which are the ZONE_AWARE configuration properties.

You can also configure these properties manually using Hazelcast's member attributes, and the following are the related property names:

hazelcast.partition.group.zone: For the zones in the same area.
hazelcast.partition.group.rack: For different racks in the same zone.
hazelcast.partition.group.host: For a shared physical member if virtualization is used.

Here is how to put them in a declarative configuration:

<member-attributes>
  <attribute name="hazelcast.partition.group.zone">zone name</attribute>
  <attribute name="hazelcast.partition.group.rack">rack name</attribute>
  <attribute name="hazelcast.partition.group.host">host name</attribute>
</member-attributes>

When using ZONE_AWARE configuration, backups are created in the other zones. Each zone will be accepted as one partition group.

NOTE: Some cloud providers have rack information instead of zone information. In such cases, Hazelcast looks for zone, rack, and host information in the given order.

Following are declarative and programmatic configuration snippets that show how to enable ZONE_AWARE grouping.

<partition-group enabled="true" group-type="ZONE_AWARE" />

Config config = ...;
PartitionGroupConfig partitionGroupConfig = config.getPartitionGroupConfig();
partitionGroupConfig.setEnabled( true )
    .setGroupType( MemberGroupType.ZONE_AWARE );

NOTE: Currently ZONE_AWARE configuration works only with Hazelcast jclouds and Hazelcast Azure Discovery Service plugins. Please refer to their GitHub repositories at Hazelcast jclouds and Hazelcast Azure for more information on these plugins.

5. SPI:

You can provide your own partition group implementation using the SPI configuration. To create your partition group implementation, you need to first extend the DiscoveryStrategy class of the discovery service plugin, override the method public PartitionGroupStrategy getPartitionGroupStrategy(), and return the PartitionGroupStrategy configuration in that overridden method.

Following is a sample code covering the implementation steps mentioned in the above paragraph:

public class CustomDiscovery extends JCloudsDiscoveryStrategy {

    public CustomDiscovery(Map<String, Comparable> properties) {
        super(properties);
    }
    
    @Override
    public PartitionGroupStrategy getPartitionGroupStrategy() {
        return new CustomPartitionGroupStrategy();
    }

    private class CustomPartitionGroupStrategy implements PartitionGroupStrategy {
        @Override
        public Iterable<MemberGroup> getMemberGroups() {
            ...
            ...
        }
    }
}

Setting Up Clusters Partition Group Configuration

Grouping Types