Uses of Interface
com.hazelcast.jet.pipeline.StreamStage
Package
Description
The Pipeline API is Jet's high-level API to build and execute
distributed computation jobs.
This package contains various mock sources to help with pipeline testing
and development.
Contributes a
PythonTransforms.mapUsingPython(com.hazelcast.jet.python.PythonServiceConfig)
transform that allows you to transform Jet pipeline data using a Python
function.-
Uses of StreamStage in com.hazelcast.jet.pipeline
Modifier and TypeMethodDescriptionGeneralStage.addTimestamps
(ToLongFunctionEx<? super T> timestampFn, long allowedLag) Adds a timestamp to each item in the stream using the supplied function and specifies the allowed amount of disorder between them.<R> StreamStage<KeyedWindowResult<K,
R>> StageWithKeyAndWindow.aggregate
(AggregateOperation1<? super T, ?, ? extends R> aggrOp) Attaches a stage that performs the given group-and-aggregate operation.<R> StreamStage<WindowResult<R>>
StageWithWindow.aggregate
(AggregateOperation1<? super T, ?, ? extends R> aggrOp) Attaches a stage that performs the given aggregate operation over all the items that belong to a given window.default <T1,
R0, R1> StreamStage<KeyedWindowResult<K, Tuple2<R0, R1>>> StageWithKeyAndWindow.aggregate2
(AggregateOperation1<? super T, ?, ? extends R0> aggrOp0, StreamStageWithKey<T1, ? extends K> stage1, AggregateOperation1<? super T1, ?, ? extends R1> aggrOp1) Attaches a stage that performs the given cogroup-and-aggregate operation over the items from both this stage andstage1
you supply.<T1,
R> StreamStage<KeyedWindowResult<K, R>> StageWithKeyAndWindow.aggregate2
(StreamStageWithKey<T1, ? extends K> stage1, AggregateOperation2<? super T, ? super T1, ?, ? extends R> aggrOp) Attaches a stage that performs the given cogroup-and-aggregate operation over the items from both this stage andstage1
you supply.default <T1,
R0, R1> StreamStage<WindowResult<Tuple2<R0, R1>>> StageWithWindow.aggregate2
(AggregateOperation1<? super T, ?, ? extends R0> aggrOp0, StreamStage<T1> stage1, AggregateOperation1<? super T1, ?, ? extends R1> aggrOp1) Attaches a stage that performs the given co-aggregate operations over the items from this stage andstage1
you supply.<T1,
R> StreamStage<WindowResult<R>> StageWithWindow.aggregate2
(StreamStage<T1> stage1, AggregateOperation2<? super T, ? super T1, ?, ? extends R> aggrOp) Attaches a stage that performs the given aggregate operation over all the items that belong to the same window.default <T1,
T2, R0, R1, R2>
StreamStage<KeyedWindowResult<K,Tuple3<R0, R1, R2>>> StageWithKeyAndWindow.aggregate3
(AggregateOperation1<? super T, ?, ? extends R0> aggrOp0, StreamStageWithKey<T1, ? extends K> stage1, AggregateOperation1<? super T1, ?, ? extends R1> aggrOp1, StreamStageWithKey<T2, ? extends K> stage2, AggregateOperation1<? super T2, ?, ? extends R2> aggrOp2) Attaches a stage that performs the given cogroup-and-aggregate operation over the items from both this stage andstage1
you supply.<T1,
T2, R> StreamStage<KeyedWindowResult<K, R>> StageWithKeyAndWindow.aggregate3
(StreamStageWithKey<T1, ? extends K> stage1, StreamStageWithKey<T2, ? extends K> stage2, AggregateOperation3<? super T, ? super T1, ? super T2, ?, ? extends R> aggrOp) Attaches a stage that performs the given cogroup-and-aggregate operation over the items from this stage as well asstage1
andstage2
you supply.default <T1,
T2, R0, R1, R2>
StreamStage<WindowResult<Tuple3<R0,R1, R2>>> StageWithWindow.aggregate3
(AggregateOperation1<? super T, ?, ? extends R0> aggrOp0, StreamStage<T1> stage1, AggregateOperation1<? super T1, ?, ? extends R1> aggrOp1, StreamStage<T2> stage2, AggregateOperation1<? super T2, ?, ? extends R2> aggrOp2) Attaches a stage that performs the given aggregate operation over all the items that belong to the same window.<T1,
T2, R> StreamStage<WindowResult<R>> StageWithWindow.aggregate3
(StreamStage<T1> stage1, StreamStage<T2> stage2, AggregateOperation3<? super T, ? super T1, ? super T2, ?, ? extends R> aggrOp) Attaches a stage that performs the given aggregate operation over the items it receives from this stage as well asstage1
andstage2
you supply.default <R> StreamStage<R>
StreamStage.apply
(FunctionEx<? super StreamStage<T>, ? extends StreamStage<R>> transformFn) Transformsthis
stage using the providedtransformFn
and returns the transformed stage.<R> StreamStage<R>
StreamHashJoinBuilder.build
(BiFunctionEx<T0, ItemsByTag, R> mapToOutputFn) Builds a new pipeline stage that performs the hash-join operation.WindowAggregateBuilder.build()
Creates and returns a pipeline stage that performs a windowed co-aggregation of the stages registered with this builder object.<A,
R> StreamStage<WindowResult<R>> WindowAggregateBuilder1.build
(AggregateOperation<A, R> aggrOp) Creates and returns a pipeline stage that performs a windowed co-aggregation of the pipeline stages registered with this builder object.WindowGroupAggregateBuilder.build()
Creates and returns a pipeline stage that performs a windowed cogroup-and-aggregate operation on the stages registered with this builder object.<R> StreamStage<KeyedWindowResult<K,
R>> WindowGroupAggregateBuilder1.build
(AggregateOperation<?, ? extends R> aggrOp) Creates and returns a pipeline stage that performs a windowed cogroup-and-aggregate of the pipeline stages registered with this builder object.default <R> StreamStage<R>
StreamStage.customTransform
(String stageName, SupplierEx<Processor> procSupplier) <R> StreamStage<R>
StreamStage.customTransform
(String stageName, ProcessorMetaSupplier procSupplier) default <R> StreamStage<R>
StreamStage.customTransform
(String stageName, ProcessorSupplier procSupplier) default <R> StreamStage<R>
StreamStageWithKey.customTransform
(String stageName, SupplierEx<Processor> procSupplier) <R> StreamStage<R>
StreamStageWithKey.customTransform
(String stageName, ProcessorMetaSupplier procSupplier) default <R> StreamStage<R>
StreamStageWithKey.customTransform
(String stageName, ProcessorSupplier procSupplier) default StreamStage<KeyedWindowResult<K,
T>> StageWithKeyAndWindow.distinct()
Attaches a stage that passes through just the items that are distinct within their window according to the grouping key (no two items emitted for a window map to the same key).default StreamStage<WindowResult<T>>
StageWithWindow.distinct()
Attaches a stage that passes through just the items that are distinct within their window (no two items emitted for a window are equal).StreamStage.filter
(PredicateEx<T> filterFn) <S> StreamStage<T>
StreamStage.filterStateful
(SupplierEx<? extends S> createFn, BiPredicateEx<? super S, ? super T> filterFn) <S> StreamStage<T>
StreamStageWithKey.filterStateful
(long ttl, SupplierEx<? extends S> createFn, BiPredicateEx<? super S, ? super T> filterFn) Attaches a stage that performs a stateful filtering operation.default <S> StreamStage<T>
StreamStageWithKey.filterStateful
(SupplierEx<? extends S> createFn, BiPredicateEx<? super S, ? super T> filterFn) <S> StreamStage<T>
StreamStage.filterUsingService
(ServiceFactory<?, S> serviceFactory, BiPredicateEx<? super S, ? super T> filterFn) <S> StreamStage<T>
StreamStageWithKey.filterUsingService
(ServiceFactory<?, S> serviceFactory, TriPredicate<? super S, ? super K, ? super T> filterFn) <R> StreamStage<R>
StreamStage.flatMap
(FunctionEx<? super T, ? extends Traverser<R>> flatMapFn) <S,
R> StreamStage<R> StreamStage.flatMapStateful
(SupplierEx<? extends S> createFn, BiFunctionEx<? super S, ? super T, ? extends Traverser<R>> flatMapFn) <S,
R> StreamStage<R> StreamStageWithKey.flatMapStateful
(long ttl, SupplierEx<? extends S> createFn, TriFunction<? super S, ? super K, ? super T, ? extends Traverser<R>> flatMapFn, TriFunction<? super S, ? super K, ? super Long, ? extends Traverser<R>> onEvictFn) Attaches a stage that performs a stateful flat-mapping operation.<S,
R> StreamStage<R> StreamStageWithKey.flatMapStateful
(SupplierEx<? extends S> createFn, TriFunction<? super S, ? super K, ? super T, ? extends Traverser<R>> flatMapFn) <S,
R> StreamStage<R> StreamStage.flatMapUsingService
(ServiceFactory<?, S> serviceFactory, BiFunctionEx<? super S, ? super T, ? extends Traverser<R>> flatMapFn) <S,
R> StreamStage<R> StreamStageWithKey.flatMapUsingService
(ServiceFactory<?, S> serviceFactory, TriFunction<? super S, ? super K, ? super T, ? extends Traverser<R>> flatMapFn) <K,
T1_IN, T1, R>
StreamStage<R>StreamStage.hashJoin
(BatchStage<T1_IN> stage1, JoinClause<K, ? super T, ? super T1_IN, ? extends T1> joinClause1, BiFunctionEx<T, T1, R> mapToOutputFn) <K1,
K2, T1_IN, T2_IN, T1, T2, R>
StreamStage<R>StreamStage.hashJoin2
(BatchStage<T1_IN> stage1, JoinClause<K1, ? super T, ? super T1_IN, ? extends T1> joinClause1, BatchStage<T2_IN> stage2, JoinClause<K2, ? super T, ? super T2_IN, ? extends T2> joinClause2, TriFunction<T, T1, T2, R> mapToOutputFn) <K,
T1_IN, T1, R>
StreamStage<R>StreamStage.innerHashJoin
(BatchStage<T1_IN> stage1, JoinClause<K, ? super T, ? super T1_IN, ? extends T1> joinClause1, BiFunctionEx<T, T1, R> mapToOutputFn) <K1,
K2, T1_IN, T2_IN, T1, T2, R>
StreamStage<R>StreamStage.innerHashJoin2
(BatchStage<T1_IN> stage1, JoinClause<K1, ? super T, ? super T1_IN, ? extends T1> joinClause1, BatchStage<T2_IN> stage2, JoinClause<K2, ? super T, ? super T2_IN, ? extends T2> joinClause2, TriFunction<T, T1, T2, R> mapToOutputFn) <R> StreamStage<R>
StreamStage.map
(FunctionEx<? super T, ? extends R> mapFn) <S,
R> StreamStage<R> StreamStage.mapStateful
(SupplierEx<? extends S> createFn, BiFunctionEx<? super S, ? super T, ? extends R> mapFn) <S,
R> StreamStage<R> StreamStageWithKey.mapStateful
(long ttl, SupplierEx<? extends S> createFn, TriFunction<? super S, ? super K, ? super T, ? extends R> mapFn, TriFunction<? super S, ? super K, ? super Long, ? extends R> onEvictFn) Attaches a stage that performs a stateful mapping operation.<S,
R> StreamStage<R> StreamStageWithKey.mapStateful
(SupplierEx<? extends S> createFn, TriFunction<? super S, ? super K, ? super T, ? extends R> mapFn) default <K,
V, R> StreamStage<R> StreamStage.mapUsingIMap
(IMap<K, V> iMap, FunctionEx<? super T, ? extends K> lookupKeyFn, BiFunctionEx<? super T, ? super V, ? extends R> mapFn) default <K,
V, R> StreamStage<R> StreamStage.mapUsingIMap
(String mapName, FunctionEx<? super T, ? extends K> lookupKeyFn, BiFunctionEx<? super T, ? super V, ? extends R> mapFn) default <V,
R> StreamStage<R> StreamStageWithKey.mapUsingIMap
(IMap<K, V> iMap, BiFunctionEx<? super T, ? super V, ? extends R> mapFn) default <V,
R> StreamStage<R> StreamStageWithKey.mapUsingIMap
(String mapName, BiFunctionEx<? super T, ? super V, ? extends R> mapFn) default <K,
V, R> StreamStage<R> StreamStage.mapUsingReplicatedMap
(ReplicatedMap<K, V> replicatedMap, FunctionEx<? super T, ? extends K> lookupKeyFn, BiFunctionEx<? super T, ? super V, ? extends R> mapFn) default <K,
V, R> StreamStage<R> StreamStage.mapUsingReplicatedMap
(String mapName, FunctionEx<? super T, ? extends K> lookupKeyFn, BiFunctionEx<? super T, ? super V, ? extends R> mapFn) <S,
R> StreamStage<R> StreamStage.mapUsingService
(ServiceFactory<?, S> serviceFactory, BiFunctionEx<? super S, ? super T, ? extends R> mapFn) <S,
R> StreamStage<R> StreamStageWithKey.mapUsingService
(ServiceFactory<?, S> serviceFactory, TriFunction<? super S, ? super K, ? super T, ? extends R> mapFn) <S,
R> StreamStage<R> StreamStage.mapUsingServiceAsync
(ServiceFactory<?, S> serviceFactory, int maxConcurrentOps, boolean preserveOrder, BiFunctionEx<? super S, ? super T, ? extends CompletableFuture<R>> mapAsyncFn) default <S,
R> StreamStage<R> StreamStage.mapUsingServiceAsync
(ServiceFactory<?, S> serviceFactory, BiFunctionEx<? super S, ? super T, ? extends CompletableFuture<R>> mapAsyncFn) <S,
R> StreamStage<R> StreamStageWithKey.mapUsingServiceAsync
(ServiceFactory<?, S> serviceFactory, int maxConcurrentOps, boolean preserveOrder, TriFunction<? super S, ? super K, ? super T, CompletableFuture<R>> mapAsyncFn) default <S,
R> StreamStage<R> StreamStageWithKey.mapUsingServiceAsync
(ServiceFactory<?, S> serviceFactory, TriFunction<? super S, ? super K, ? super T, CompletableFuture<R>> mapAsyncFn) <S,
R> StreamStage<R> StreamStage.mapUsingServiceAsyncBatched
(ServiceFactory<?, S> serviceFactory, int maxBatchSize, BiFunctionEx<? super S, ? super List<T>, ? extends CompletableFuture<List<R>>> mapAsyncFn) <S,
R> StreamStage<R> StreamStageWithKey.mapUsingServiceAsyncBatched
(ServiceFactory<?, S> serviceFactory, int maxBatchSize, BiFunctionEx<? super S, ? super List<T>, ? extends CompletableFuture<List<R>>> mapAsyncFn) <S,
R> StreamStage<R> StreamStageWithKey.mapUsingServiceAsyncBatched
(ServiceFactory<?, S> serviceFactory, int maxBatchSize, TriFunction<? super S, ? super List<K>, ? super List<T>, ? extends CompletableFuture<List<R>>> mapAsyncFn) StreamStage.merge
(StreamStage<? extends T> other) Attaches a stage that emits all the items from this stage as well as all the items from the supplied stage.default StreamStage<T>
StreamStage.peek()
default StreamStage<T>
StreamStage.peek
(FunctionEx<? super T, ? extends CharSequence> toStringFn) StreamStage.peek
(PredicateEx<? super T> shouldLogFn, FunctionEx<? super T, ? extends CharSequence> toStringFn) StreamStage.rebalance()
<K> StreamStage<T>
StreamStage.rebalance
(FunctionEx<? super T, ? extends K> keyFn) default <A,
R> StreamStage<R> StreamStage.rollingAggregate
(AggregateOperation1<? super T, A, ? extends R> aggrOp) default <A,
R> StreamStage<Map.Entry<K, R>> StreamStageWithKey.rollingAggregate
(long ttl, AggregateOperation1<? super T, A, ? extends R> aggrOp) Attaches a rolling aggregation stage.default <A,
R> StreamStage<Map.Entry<K, R>> StreamStageWithKey.rollingAggregate
(AggregateOperation1<? super T, A, ? extends R> aggrOp) StreamStage.setLocalParallelism
(int localParallelism) StageWithWindow.streamStage()
Returns the pipeline stage associated with this object.StreamSourceStage.withIngestionTimestamps()
Declares that the source will assign the time of ingestion as the event timestamp.StreamSourceStage.withNativeTimestamps
(long allowedLag) Declares that the stream will use the source's native timestamps.StreamSourceStage.withoutTimestamps()
Declares that the source will not assign any timestamp to the events it emits.StreamSourceStage.withTimestamps
(ToLongFunctionEx<? super T> timestampFn, long allowedLag) Declares that the source will extract timestamps from the stream items.Modifier and TypeMethodDescription<T,
R> Tag<R> WindowAggregateBuilder.add
(StreamStage<T> stage, AggregateOperation1<? super T, ?, ? extends R> aggrOp) Adds another stage that will contribute its data to the aggregate operation to be performed.<E> Tag<E>
WindowAggregateBuilder1.add
(StreamStage<E> stage) Adds another stage that will contribute its data to the aggregate operation to be performed.default <T1,
R0, R1> StreamStage<WindowResult<Tuple2<R0, R1>>> StageWithWindow.aggregate2
(AggregateOperation1<? super T, ?, ? extends R0> aggrOp0, StreamStage<T1> stage1, AggregateOperation1<? super T1, ?, ? extends R1> aggrOp1) Attaches a stage that performs the given co-aggregate operations over the items from this stage andstage1
you supply.<T1,
R> StreamStage<WindowResult<R>> StageWithWindow.aggregate2
(StreamStage<T1> stage1, AggregateOperation2<? super T, ? super T1, ?, ? extends R> aggrOp) Attaches a stage that performs the given aggregate operation over all the items that belong to the same window.default <T1,
T2, R0, R1, R2>
StreamStage<WindowResult<Tuple3<R0,R1, R2>>> StageWithWindow.aggregate3
(AggregateOperation1<? super T, ?, ? extends R0> aggrOp0, StreamStage<T1> stage1, AggregateOperation1<? super T1, ?, ? extends R1> aggrOp1, StreamStage<T2> stage2, AggregateOperation1<? super T2, ?, ? extends R2> aggrOp2) Attaches a stage that performs the given aggregate operation over all the items that belong to the same window.<T1,
T2, R> StreamStage<WindowResult<R>> StageWithWindow.aggregate3
(StreamStage<T1> stage1, StreamStage<T2> stage2, AggregateOperation3<? super T, ? super T1, ? super T2, ?, ? extends R> aggrOp) Attaches a stage that performs the given aggregate operation over the items it receives from this stage as well asstage1
andstage2
you supply.StreamStage.merge
(StreamStage<? extends T> other) Attaches a stage that emits all the items from this stage as well as all the items from the supplied stage.Modifier and TypeMethodDescriptiondefault <R> StreamStage<R>
StreamStage.apply
(FunctionEx<? super StreamStage<T>, ? extends StreamStage<R>> transformFn) Transformsthis
stage using the providedtransformFn
and returns the transformed stage.default <R> StreamStage<R>
StreamStage.apply
(FunctionEx<? super StreamStage<T>, ? extends StreamStage<R>> transformFn) Transformsthis
stage using the providedtransformFn
and returns the transformed stage. -
Uses of StreamStage in com.hazelcast.jet.pipeline.test
Modifier and TypeMethodDescriptionstatic <T> FunctionEx<StreamStage<T>,
StreamStage<T>> Assertions.assertCollectedEventually
(int timeout, ConsumerEx<? super List<T>> assertFn) Collects all the received items into a list and runs theassertFn
every time a new item is received.static <T> FunctionEx<StreamStage<T>,
StreamStage<T>> Assertions.assertCollectedEventually
(int timeout, ConsumerEx<? super List<T>> assertFn) Collects all the received items into a list and runs theassertFn
every time a new item is received. -
Uses of StreamStage in com.hazelcast.jet.python
Modifier and TypeMethodDescriptionstatic <K> FunctionEx<StreamStage<String>,
StreamStage<String>> PythonTransforms.mapUsingPython
(FunctionEx<? super String, ? extends K> keyFn, PythonServiceConfig cfg) Deprecated.static <K> FunctionEx<StreamStage<String>,
StreamStage<String>> PythonTransforms.mapUsingPython
(FunctionEx<? super String, ? extends K> keyFn, PythonServiceConfig cfg) Deprecated.Jet now has first-class support for data rebalancing, seeGeneralStage.rebalance()
andGeneralStage.rebalance(FunctionEx)
.static FunctionEx<StreamStage<String>,
StreamStage<String>> PythonTransforms.mapUsingPython
(PythonServiceConfig cfg) A stage-transforming method that adds a "map using Python" pipeline stage.static FunctionEx<StreamStage<String>,
StreamStage<String>> PythonTransforms.mapUsingPython
(PythonServiceConfig cfg) A stage-transforming method that adds a "map using Python" pipeline stage.static FunctionEx<StreamStage<String>,
StreamStage<String>> PythonTransforms.mapUsingPython
(PythonServiceConfig cfg, int maxBatchSize) A stage-transforming method that adds a "map using Python" pipeline stage.static FunctionEx<StreamStage<String>,
StreamStage<String>> PythonTransforms.mapUsingPython
(PythonServiceConfig cfg, int maxBatchSize) A stage-transforming method that adds a "map using Python" pipeline stage.
GeneralStage.rebalance()
andGeneralStage.rebalance(FunctionEx)
.