Package com.hazelcast.jet.pipeline

Class RemoteMapSourceBuilder<K,V,T>

java.lang.Object
com.hazelcast.jet.pipeline.RemoteMapSourceBuilder<K,V,T>
Type Parameters:
K - the type of the key in the map
V - the type of the value in the map
T - the type of the emitted items
public class RemoteMapSourceBuilder<K,V,T> extends Object
Builder providing a fluent API to build a remote map source. Use Sources.remoteMapBuilder(String) to obtain the builder instance.

By supplying a predicate and projection here instead of in separate map/filter transforms you allow the source to apply these functions early, before generating any output, with the potential of significantly reducing data traffic. If your data is stored in the IMDG using the portable serialization format, there are additional optimizations available when using Projections.singleAttribute(java.lang.String) and Projections.multiAttribute(java.lang.String...)) to create your projection instance and using the Predicates factory or PredicateBuilder to create the predicate. In this case Jet can test the predicate and apply the projection without deserializing the whole object.

Due to the current limitations in the way Jet reads the map it can't use any indexes on the map. It will always scan the map in full.

The source does not save any state to snapshot. If the job is restarted, it will re-emit all entries.

If the IMap is modified while being read, or if there is a cluster topology change (triggering data migration), the source may miss and/or duplicate some entries. If we detect a topology change, the job will fail, but the detection is only on a best-effort basis - we might still give incorrect results without reporting a failure. Concurrent mutation is not detected at all.

The default local parallelism for this processor is 1.

Predicate/projection class requirements

The classes implementing predicate and projection need to be available on the remote cluster's classpath or loaded using Hazelcast User Code Deployment. It's not enough to add them to the job classpath in JobConfig. The same is true for the class of the objects stored in the map itself. If you cannot meet these conditions, do not add predicate and projection and add a subsequent map or filter stage.

Since:
5.4

  • Method Details

    • dataConnectionRef

      public RemoteMapSourceBuilder<K,V,T> dataConnectionRef(@Nonnull DataConnectionRef dataConnectionRef)
      Set the data connection name to use to connect to the remote cluster. The data connection must be of HazelcastDataConnection type.

      One of dataConnectionRef or clientConfig is required to build the source. If both are provided the data connection takes precedence.

      Parameters:
      dataConnectionRef - name of the data connection
      Returns:
      this builder

    • clientConfig

      public RemoteMapSourceBuilder<K,V,T> clientConfig(@Nonnull ClientConfig clientConfig)
      Set the client configuration to use to connect to the remote cluster.

      One of dataConnectionRef or clientConfig is required to build the source. If both are provided the data connection takes precedence.

      Parameters:
      clientConfig - client configuration
      Returns:
      this builder

    • predicate

      public RemoteMapSourceBuilder<K,V,T> predicate(@Nonnull Predicate<K,V> predicate)
      Set the predicate to apply at the source. See the RemoteMapSourceBuilder for more details.
      Parameters:
      predicate - the predicate
      Returns:
      this builder

    • projection

      public <T_NEW> RemoteMapSourceBuilder<K,V,T_NEW> projection(@Nonnull Projection<? super Map.Entry<K,V>,? extends T_NEW> projection)
      Set the projection to apply at the source. See the RemoteMapSourceBuilder for more details.
      Type Parameters:
      T_NEW - type of the emitted items
      Parameters:
      projection - the projection
      Returns:
      this builder

    • build

      public <N> BatchSource<N> build()
      Build the source using the parameters set in this builder.
      Returns:
      a batch source emitting items from the remote IMap