Package com.hazelcast.jet.pipeline

Interface IMapExtension.IMapGeneralStage<T,S extends IMapExtension.IMapGeneralStage<T,S>>

Type Parameters:
T - type of the input item
S - type of this stage
All Known Subinterfaces:
IMapExtension.IMapBatchStage<T>, IMapExtension.IMapStreamStage<T>
Enclosing interface:
IMapExtension<T,K>
public static interface IMapExtension.IMapGeneralStage<T,S extends IMapExtension.IMapGeneralStage<T,S>>
Extends Pipeline stage with convenient methods utilising IMap in addition to GeneralStage.mapUsingIMap(java.lang.String, com.hazelcast.function.FunctionEx<? super T, ? extends K>, com.hazelcast.function.BiFunctionEx<? super T, ? super V, ? extends R>) available in basic stages.
Since:
5.7

  • Method Summary

    Modifier and Type
    Method
    Description
    S
    doNotPreserveOrder()
    Disables preservation of ordering of the input items by this stage.
    default <K, V, R> com.hazelcast.jet.pipeline.GeneralStage<R>
    mapUsingPutIfAbsent(com.hazelcast.map.IMap<K,V> iMap, com.hazelcast.function.FunctionEx<? super T,? extends K> lookupKeyFn, com.hazelcast.function.FunctionEx<? super T,? extends V> newValueFn, com.hazelcast.jet.function.TriFunction<? super T,? super V,? super V,? extends R> mapFn)
    Attaches a mapping stage where for each item a IMap.putIfAbsent(K, V) operation is performed in the supplied IMap.
    <K, V, R> com.hazelcast.jet.pipeline.GeneralStage<R>
    mapUsingPutIfAbsent(String mapName, com.hazelcast.function.FunctionEx<? super T,? extends K> lookupKeyFn, com.hazelcast.function.FunctionEx<? super T,? extends V> newValueFn, com.hazelcast.jet.function.TriFunction<? super T,? super V,? super V,? extends R> mapFn)
    Attaches a mapping stage where for each item a IMap.putIfAbsent(K, V) operation is performed in the IMap with the supplied name.
    S
    maxConcurrentOps(int maxConcurrentOps)
    Sets maximum number of concurrent async operations to be used by this stage.

  • Method Details

    • mapUsingPutIfAbsent

      @Nonnull <K, V, R> com.hazelcast.jet.pipeline.GeneralStage<R> mapUsingPutIfAbsent(@Nonnull String mapName, @Nonnull com.hazelcast.function.FunctionEx<? super T,? extends K> lookupKeyFn, @Nonnull com.hazelcast.function.FunctionEx<? super T,? extends V> newValueFn, @Nonnull com.hazelcast.jet.function.TriFunction<? super T,? super V,? super V,? extends R> mapFn)
      Attaches a mapping stage where for each item a IMap.putIfAbsent(K, V) operation is performed in the IMap with the supplied name. The result of IMap.putIfAbsent(K, V) (previous value associated with the key or null) is merged with the item and emitted.

      If the result of the mapping is null, it emits nothing. Therefore, this stage can be used to implement filtering semantics as well.

      The mapping logic is equivalent to: 

      
 K key = lookupKeyFn.apply(item);
 V newValue = newValueFn.apply(item);
 V prevValue = map.putIfAbsent(key, newValue);
 return mapFn.apply(item, prevValue, newValue);

      This operation and the IMap do not participate in Jet's fault tolerance protocol. If upstream stage or source retries processing, it will be invoked again. However, keep in mind that IMap.putIfAbsent(K, V) is not idempotent: when retried it may produce a different result (previous value) than in the original invocation. The mapping function and pipeline logic must be able to deal with this.

      This method does not use transactions for IMap. All updates to IMap are immediately visible regardless of Jet's fault tolerance protocol (snapshots).

      Type Parameters:
      K - type of the key in the IMap
      V - type of the value in the IMap
      R - type of the output item
      Parameters:
      mapName - name of the IMap
      lookupKeyFn - a function which returns the IMap key. Must not return null. It must be stateless and cooperative.
      newValueFn - a function which returns the IMap value to be associated with the key. Must not return null. It must be stateless and cooperative.
      mapFn - the mapping function. It must be stateless and cooperative.
      Returns:
      the newly attached stage

    • mapUsingPutIfAbsent

      @Nonnull default <K, V, R> com.hazelcast.jet.pipeline.GeneralStage<R> mapUsingPutIfAbsent(@Nonnull com.hazelcast.map.IMap<K,V> iMap, @Nonnull com.hazelcast.function.FunctionEx<? super T,? extends K> lookupKeyFn, @Nonnull com.hazelcast.function.FunctionEx<? super T,? extends V> newValueFn, @Nonnull com.hazelcast.jet.function.TriFunction<? super T,? super V,? super V,? extends R> mapFn)
      Attaches a mapping stage where for each item a IMap.putIfAbsent(K, V) operation is performed in the supplied IMap. The result of IMap.putIfAbsent(K, V) (previous value associated with the key or null) is merged with the item and emitted.

      If the result of the mapping is null, it emits nothing. Therefore, this stage can be used to implement filtering semantics as well.

      The mapping logic is equivalent to: 

      
 K key = lookupKeyFn.apply(item);
 V newValue = newValueFn.apply(item);
 V prevValue = map.putIfAbsent(key, newValue);
 return mapFn.apply(item, prevValue, newValue);

      This operation and the IMap do not participate in Jet's fault tolerance protocol. If upstream stage or source retries processing, it will be invoked again. However, keep in mind that IMap.putIfAbsent(K, V) is not idempotent: when retried it may produce a different result (previous value) than in the original invocation. The mapping function and pipeline logic must be able to deal with this.

      This method does not use transactions for IMap. All updates to IMap are immediately visible regardless of Jet's fault tolerance protocol (snapshots).

      Type Parameters:
      K - type of the key in the IMap
      V - type of the value in the IMap
      R - type of the output item
      Parameters:
      iMap - the IMap to use
      lookupKeyFn - a function which returns the IMap key. Must not return null. It must be stateless and cooperative.
      newValueFn - a function which returns the IMap value to be associated with the key. Must not return null. It must be stateless and cooperative.
      mapFn - the mapping function. It must be stateless and cooperative.
      Returns:
      the newly attached stage

    • maxConcurrentOps

      @Nonnull S maxConcurrentOps(int maxConcurrentOps)
      Sets maximum number of concurrent async operations to be used by this stage. Jet will execute at most this many concurrent async operations per processor and will apply backpressure to the upstream to enforce it.

      Default value is 4.

      Parameters:
      maxConcurrentOps - maximum number of concurrent async operations per processor
      Returns:
      this stage
      See Also:
      • GeneralStage.mapUsingServiceAsync(ServiceFactory, int, boolean, BiFunctionEx)

    • doNotPreserveOrder

      @Nonnull S doNotPreserveOrder()
      Disables preservation of ordering of the input items by this stage.

      Jet can process asynchronous responses in two modes:

      1. Ordered: results of the async calls are emitted in the submission order. This is the default.
      2. Unordered: results of the async calls are emitted as they arrive. This mode is enabled by this method.
      The unordered mode can be faster:
      • in the ordered mode, one stalling call will block all subsequent items, even though responses for them were already received
      • to preserve the order after a restart, the ordered implementation when saving the state to the snapshot waits for all async calls to complete. This creates a hiccup depending on the async call latency. The unordered one saves in-flight items to the state snapshot.
      The order of watermarks is preserved even in the unordered mode. Jet forwards the watermark after having emitted all the results of the items that came before it. One stalling response will prevent a windowed operation downstream from finishing, but if the operation is configured to emit early results, they will be more correct with the unordered approach.
      Returns:
      this stage
      See Also:
      • GeneralStage.mapUsingServiceAsync(ServiceFactory, int, boolean, BiFunctionEx)