Package com.hazelcast.jet.pipeline

Class EnterpriseSinks

java.lang.Object
com.hazelcast.jet.pipeline.EnterpriseSinks
public class EnterpriseSinks extends Object
Contains factory methods for enterprise-only pipeline sinks.
Since:
5.7

  • Method Summary

    Modifier and Type
    Method
    Description
    static <T> com.hazelcast.jet.pipeline.Sink<T>
    mapFlushSink(com.hazelcast.map.IMap<?,?> map, boolean isLocalFlush)
    Returns a sink that flushes entries from the specified IMap to its configured MapStore as part of the Hazelcast Jet snapshotting process.
    static <T> com.hazelcast.jet.pipeline.Sink<T>
    mapFlushSink(com.hazelcast.map.IMap<?,?> map, boolean isLocalFlush, Duration flushTimeout)
    Returns a sink that flushes entries from the specified IMap to its configured MapStore as part of the Hazelcast Jet snapshotting process.
    static <T> com.hazelcast.jet.pipeline.Sink<T>
    mapFlushSink(String mapName, boolean isLocalFlush)
    Returns a sink that flushes entries from the specified IMap to its configured MapStore as part of the Hazelcast Jet snapshotting process.
    static <T> com.hazelcast.jet.pipeline.Sink<T>
    mapFlushSink(String mapName, boolean isLocalFlush, Duration flushTimeout)
    Returns a sink that flushes entries from the specified IMap to its configured MapStore as part of the Hazelcast Jet snapshotting process.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • mapFlushSink

      @Nonnull public static <T> com.hazelcast.jet.pipeline.Sink<T> mapFlushSink(@Nonnull String mapName, boolean isLocalFlush)
      Returns a sink that flushes entries from the specified IMap to its configured MapStore as part of the Hazelcast Jet snapshotting process.

      The sink ensures that all pending write-behind updates are flushed to the underlying storage before a job snapshot is committed.

      The sink can also be used in batch jobs to flush the MapStore when the job completes, ensuring that all data is persisted before termination.

      The sink supports two flushing modes:

      • Local flush (isLocalFlush == true): each processor instance flushes only the partitions owned by the current member. This mode is more efficient but requires that the upstream pipeline processes updates locally and does not repartition or rebalance items before reaching this sink.
      • Global flush (isLocalFlush == false): all partitions across all members are flushed. This mode is less efficient but safe when local processing guarantees cannot be ensured.

      Local flush mode must be used only when it is guaranteed that the partitions updated by upstream processors exactly match the partitions flushed by this sink. Otherwise, mismatched partitions may lead to data loss in case of failures.

      By default, the system waits indefinitely for the flush operation to complete. To modify this behavior, use the appropriate methods that accept a flush timeout: mapFlushSink(String, boolean, Duration)}, mapFlushSink(IMap, boolean, Duration).}

      Example usage: 
      
 // target-map has write-behind MapStore configured
 p.readFrom(Sources.map("source-imap"))
     // Rebalance only if required for correct partitioning
     .rebalance(Map.Entry::getKey)
     .mapUsingServiceAsync(ServiceFactories.iMapService("target-map"),
         (map, entry) ->
             map.putAsync(entry.getKey(), entry.getValue()).toCompletableFuture())
     .writeTo(mapFlushSink("target-map", true));

      Note that local parallelism of 1 is sufficient, since this sink does not perform meaningful item processing and only invokes IMap.flush() during snapshotting.

      Type Parameters:
      T - the type of items accepted by the sink
      Parameters:
      mapName - the name of the IMap to flush
      isLocalFlush - true to flush only partitions owned by the current processor, false to flush all partitions globally
      Returns:
      a sink that flushes the specified IMap to its MapStore
      Since:
      5.7

    • mapFlushSink

      @Nonnull public static <T> com.hazelcast.jet.pipeline.Sink<T> mapFlushSink(@Nonnull com.hazelcast.map.IMap<?,?> map, boolean isLocalFlush)
      Returns a sink that flushes entries from the specified IMap to its configured MapStore as part of the Hazelcast Jet snapshotting process.

      This method is the same as the mapFlushSink(String, boolean)} method, but allows to pass the IMap directly.

      Type Parameters:
      T - the type of items accepted by the sink
      Parameters:
      map - the IMap to flush
      isLocalFlush - true to flush only partitions owned by the current processor, false to flush all partitions globally
      Returns:
      a sink that flushes the specified IMap to its MapStore
      Since:
      5.7

    • mapFlushSink

      @Nonnull public static <T> com.hazelcast.jet.pipeline.Sink<T> mapFlushSink(@Nonnull String mapName, boolean isLocalFlush, @Nonnull Duration flushTimeout)
      Returns a sink that flushes entries from the specified IMap to its configured MapStore as part of the Hazelcast Jet snapshotting process.

      This method is the same as the mapFlushSink(String, boolean)} method, but allows to pass flush timeout. flushTimeout determines the maximum time the system waits for the flush operation to finish. If the timeout is reached, the system logs a warning and continues processing without throwing an exception.

      Type Parameters:
      T - the type of items accepted by the sink
      Parameters:
      mapName - the name of the IMap to flush
      isLocalFlush - true to flush only partitions owned by the current processor, false to flush all partitions globally
      flushTimeout - the maximum time to wait for the flush
      Returns:
      a sink that flushes the specified IMap to its MapStore
      Since:
      5.7

    • mapFlushSink

      @Nonnull public static <T> com.hazelcast.jet.pipeline.Sink<T> mapFlushSink(@Nonnull com.hazelcast.map.IMap<?,?> map, boolean isLocalFlush, Duration flushTimeout)
      Returns a sink that flushes entries from the specified IMap to its configured MapStore as part of the Hazelcast Jet snapshotting process.

      The same as the mapFlushSink(String, boolean)} method, but allows to pass the IMap directly and configure flush timeout. flushTimeout determines the maximum time the system waits for the flush operation to finish. If the timeout is reached, the system logs a warning and continues processing without throwing an exception.

      Type Parameters:
      T - the type of items accepted by the sink
      Parameters:
      map - the IMap to flush
      isLocalFlush - true to flush only partitions owned by the current processor, false to flush all partitions globally
      flushTimeout - the maximum time to wait for the flush
      Returns:
      a sink that flushes the specified IMap to its MapStore
      Since:
      5.7