com.hazelcast.jet

Interface ComputeStage<E>

Type Parameters:

E - the type of items coming out of this stage

All Superinterfaces:

Stage

public interface ComputeStage<E>
extends Stage

Represents a stage in a distributed computation pipeline. It accepts input from its upstream stages (if any) and passes its output to its downstream stages.

Method Summary

Modifier and Type	Method and Description
<K,A,E1,R> ComputeStage<Map.Entry<K,R>>	coGroup(DistributedFunction<? super E,? extends K> thisKeyFn, ComputeStage<E1> stage1, DistributedFunction<? super E1,? extends K> key1Fn, AggregateOperation2<? super E,? super E1,A,R> aggrOp) Attaches to this and the supplied stage a stage that co-groups their items by a common key and applies the supplied aggregate operation to co-grouped items.
<K,A,E1,E2,R> ComputeStage<Map.Entry<K,R>>	coGroup(DistributedFunction<? super E,? extends K> thisKeyFn, ComputeStage<E1> stage1, DistributedFunction<? super E1,? extends K> key1Fn, ComputeStage<E2> stage2, DistributedFunction<? super E2,? extends K> key2Fn, AggregateOperation3<? super E,? super E1,? super E2,A,R> aggrOp) Attaches to this and the supplied stages a stage that co-groups their items by a common key and applies the supplied aggregate operation to co-grouped items.
default <K> CoGroupBuilder<K,E>	coGroupBuilder(DistributedFunction<? super E,K> thisKeyFn) Returns a fluent API builder object to construct a co-group operation with any number of contributing stages.
<R> ComputeStage<R>	customTransform(String stageName, DistributedSupplier<Processor> procSupplier) Attaches to this stage a stage with a custom transform based on the provided supplier of Core API Processors.
SinkStage	drainTo(Sink<? super E> sink) Attaches to this stage a sink stage, one that accepts data but doesn't emit any.
ComputeStage<E>	filter(DistributedPredicate<E> filterFn) Attaches to this stage a filtering stage, one which applies the provided predicate function to each input item to decide whether to pass the item to the output or to discard it.
<R> ComputeStage<R>	flatMap(DistributedFunction<? super E,Traverser<? extends R>> flatMapFn) Attaches to this stage a flat-mapping stage, one which applies the supplied function to each input item independently and emits all items from the Traverser it returns as the output items.
<K,A,R> ComputeStage<Map.Entry<K,R>>	groupBy(DistributedFunction<? super E,? extends K> keyFn, AggregateOperation1<? super E,A,R> aggrOp) Attaches to this stage a group-by-key stage, one which will group all received items by the key returned from the provided key-extracting function.
<K1,E1_IN,E1,K2,E2_IN,E2> ComputeStage<Tuple3<E,E1,E2>>	hashJoin(ComputeStage<E1_IN> stage1, JoinClause<K1,E,E1_IN,E1> joinClause1, ComputeStage<E2_IN> stage2, JoinClause<K2,E,E2_IN,E2> joinClause2) Attaches to this and the two supplied stages a hash-joining stage and returns it.
<K,E1_IN,E1> ComputeStage<Tuple2<E,E1>>	hashJoin(ComputeStage<E1_IN> stage1, JoinClause<K,E,E1_IN,E1> joinClause) Attaches to both this and the supplied stage a hash-joining stage and returns it.
default HashJoinBuilder<E>	hashJoinBuilder() Returns a fluent API builder object to construct a hash join operation with any number of contributing stages.
<R> ComputeStage<R>	map(DistributedFunction<? super E,? extends R> mapFn) Attaches to this stage a mapping stage, one which applies the supplied function to each input item independently and emits the function's result as the output item.
default ComputeStage<E>	peek() Adds a peeking layer to this compute stage which logs its output.
default ComputeStage<E>	peek(DistributedFunction<? super E,String> toStringFn) Adds a peeking layer to this compute stage which logs its output.
ComputeStage<E>	peek(DistributedPredicate<? super E> shouldLogFn, DistributedFunction<? super E,String> toStringFn) Adds a peeking layer to this compute stage which logs its output.

Methods inherited from interface com.hazelcast.jet.Stage

getPipeline

Method Detail

map

<R> ComputeStage<R> map(DistributedFunction<? super E,? extends R> mapFn)

Attaches to this stage a mapping stage, one which applies the supplied function to each input item independently and emits the function's result as the output item. Returns the newly attached stage.

Type Parameters:

R - the result type of the mapping function

Parameters:

mapFn - the mapping function

filter

ComputeStage<E> filter(DistributedPredicate<E> filterFn)

Attaches to this stage a filtering stage, one which applies the provided predicate function to each input item to decide whether to pass the item to the output or to discard it. Returns the newly attached stage.

Parameters:

filterFn - the filter predicate function

flatMap

<R> ComputeStage<R> flatMap(DistributedFunction<? super E,Traverser<? extends R>> flatMapFn)

Attaches to this stage a flat-mapping stage, one which applies the supplied function to each input item independently and emits all items from the Traverser it returns as the output items. Returns the newly attached stage.

Type Parameters:

R - the type of items in the result's traversers

Parameters:

flatMapFn - the flatmapping function, whose result type is Jet's Traverser

groupBy

<K,A,R> ComputeStage<Map.Entry<K,R>> groupBy(DistributedFunction<? super E,? extends K> keyFn,
                                             AggregateOperation1<? super E,A,R> aggrOp)

Attaches to this stage a group-by-key stage, one which will group all received items by the key returned from the provided key-extracting function. It will apply the provided aggregate operation to the items in each group and emit the result of aggregation per grouping key as the results.

Type Parameters:

K - the type of key

A - the type of the accumulator

R - the type of the aggregation result

Parameters:

keyFn - the function that extracts the grouping key from an item

aggrOp - the aggregate operation to perform

hashJoin

<K,E1_IN,E1> ComputeStage<Tuple2<E,E1>> hashJoin(ComputeStage<E1_IN> stage1,
                                                 JoinClause<K,E,E1_IN,E1> joinClause)

Attaches to both this and the supplied stage a hash-joining stage and returns it. This stage plays the role of the primary stage in the hash-join. Please refer to the package Javadoc for a detailed description of the hash-join transform.

Type Parameters:

K - the type of the join key

E1_IN - the type of stage1 items

E1 - the result type of projection on stage1 items

Parameters:

stage1 - the stage to hash-join with this one

joinClause - specifies how to join the two streams

hashJoin

<K1,E1_IN,E1,K2,E2_IN,E2> ComputeStage<Tuple3<E,E1,E2>> hashJoin(ComputeStage<E1_IN> stage1,
                                                                 JoinClause<K1,E,E1_IN,E1> joinClause1,
                                                                 ComputeStage<E2_IN> stage2,
                                                                 JoinClause<K2,E,E2_IN,E2> joinClause2)

Attaches to this and the two supplied stages a hash-joining stage and returns it. This stage plays the role of the primary stage in the hash-join. Please refer to the package Javadoc for a detailed description of the hash-join transform.

Type Parameters:

K1 - the type of key for stage1

E1_IN - the type of stage1 items

E1 - the result type of projection of stage1 items

K2 - the type of key for stage2

E2_IN - the type of stage2 items

E2 - the result type of projection of stage2 items

Parameters:

stage1 - the first stage to join

joinClause1 - specifies how to join with stage1

stage2 - the second stage to join

joinClause2 - specifices how to join with stage2

hashJoinBuilder

default HashJoinBuilder<E> hashJoinBuilder()

Returns a fluent API builder object to construct a hash join operation with any number of contributing stages. This object is mainly intended to build a hash-join of the primary stage with three or more contributing stages. For one or two stages the direct stage.hashJoin(...) calls should be preferred because they offer more static type safety.

coGroup

<K,A,E1,R> ComputeStage<Map.Entry<K,R>> coGroup(DistributedFunction<? super E,? extends K> thisKeyFn,
                                                ComputeStage<E1> stage1,
                                                DistributedFunction<? super E1,? extends K> key1Fn,
                                                AggregateOperation2<? super E,? super E1,A,R> aggrOp)

Attaches to this and the supplied stage a stage that co-groups their items by a common key and applies the supplied aggregate operation to co-grouped items.

Type Parameters:

K - the type of the grouping key

A - the type of the accumulator

E1 - the type of stage1 items

R - the result type of the aggregate operation

Parameters:

thisKeyFn - a function that extracts the grouping key from this stage's items

stage1 - the stage to co-group with this one

key1Fn - a function that extracts the grouping key from stage1 items

aggrOp - the aggregate operation to perform on co-grouped items

coGroup

<K,A,E1,E2,R> ComputeStage<Map.Entry<K,R>> coGroup(DistributedFunction<? super E,? extends K> thisKeyFn,
                                                   ComputeStage<E1> stage1,
                                                   DistributedFunction<? super E1,? extends K> key1Fn,
                                                   ComputeStage<E2> stage2,
                                                   DistributedFunction<? super E2,? extends K> key2Fn,
                                                   AggregateOperation3<? super E,? super E1,? super E2,A,R> aggrOp)

Attaches to this and the supplied stages a stage that co-groups their items by a common key and applies the supplied aggregate operation to co-grouped items.

Type Parameters:

K - the type of the grouping key

A - the type of the accumulator

E1 - the type of stage1 items

E2 - the type of stage1 items

R - the result type of the aggregate operation

Parameters:

thisKeyFn - a function that extracts the grouping key from this stage's items

stage1 - the first stage to co-group with this one

key1Fn - a function that extracts the grouping key from stage1 items

stage2 - the second stage to co-group with this one

key2Fn - a function that extracts the grouping key from stage2 items

aggrOp - the aggregate operation to perform on co-grouped items

coGroupBuilder

default <K> CoGroupBuilder<K,E> coGroupBuilder(DistributedFunction<? super E,K> thisKeyFn)

Returns a fluent API builder object to construct a co-group operation with any number of contributing stages. This object is mainly intended to build a co-grouping of the primary stage with three or more contributing stages. For one or two stages the direct stage.coGroup(...) calls should be preferred because they offer more static type safety.

Type Parameters:

K - the type of the grouping key

Parameters:

thisKeyFn - a function that extracts the grouping key from this stage's items

peek

ComputeStage<E> peek(@Nonnull
                     DistributedPredicate<? super E> shouldLogFn,
                     @Nonnull
                     DistributedFunction<? super E,String> toStringFn)

Adds a peeking layer to this compute stage which logs its output. For each item the stage emits, it:

1. uses the shouldLogFn predicate to see whether to log the item
2. if the item passed, uses toStringFn to get a string representation of the item
3. logs the string at the INFO level to the log category com.hazelcast.jet.impl.processor.PeekWrappedP.<vertexName>#<processorIndex>

The stage logs each item on whichever cluster member it happens to receive it. Its primary purpose is for development use, when running Jet on a local machine.

Parameters:

shouldLogFn - a function to filter the logged items. You can use alwaysTrue() as a pass-through filter when you don't need any filtering.

toStringFn - a function that returns a string representation of the item

See Also:

peek(DistributedFunction), peek()

peek

default ComputeStage<E> peek(@Nonnull
                             DistributedFunction<? super E,String> toStringFn)

Adds a peeking layer to this compute stage which logs its output. For each item the stage emits, it:

1. uses toStringFn to get a string representation of the item
2. logs the string at the INFO level to the log category com.hazelcast.jet.impl.processor.PeekWrappedP.<vertexName>#<processorIndex>

The stage logs each item on whichever cluster member it happens to receive it. Its primary purpose is for development use, when running Jet on a local machine.

Parameters:

toStringFn - a function that returns a string representation of the item

See Also:

peek(DistributedPredicate, DistributedFunction), peek()

peek

default ComputeStage<E> peek()

Adds a peeking layer to this compute stage which logs its output. For each item the stage emits, it logs the result of its toString() method at the INFO level to the log category com.hazelcast.jet.impl.processor.PeekWrappedP.<vertexName>#<processorIndex>. The stage logs each item on whichever cluster member it happens to receive it. Its primary purpose is for development use, when running Jet on a local machine.

See Also:

peek(DistributedPredicate, DistributedFunction), peek(DistributedFunction)

drainTo

SinkStage drainTo(Sink<? super E> sink)

Attaches to this stage a sink stage, one that accepts data but doesn't emit any. The supplied argument specifies what to do with the received data (typically push it to some outside resource).

customTransform

<R> ComputeStage<R> customTransform(String stageName,
                                    DistributedSupplier<Processor> procSupplier)

Attaches to this stage a stage with a custom transform based on the provided supplier of Core API Processors. To be compatible with the rest of the pipeline, the processor must expect a single inbound edge and arbitrarily many outbound edges, and it must push the same data to all outbound edges.

Note that the returned stage's type parameter is inferred from the call site and not propagated from the processor that will produce the result, so there is no actual type safety provided.

Type Parameters:

R - the type of the output items

Parameters:

stageName - a human-readable name for the custom stage

procSupplier - the supplier of processors