com.hazelcast.jet.pipeline

Class SinkBuilder<C,T>

java.lang.Object

com.hazelcast.jet.pipeline.SinkBuilder<C,T>

Type Parameters:

C - type of the context object

T - type of the items the sink will accept

public final class SinkBuilder<C,T>
extends Object

See sinkBuilder(String, FunctionEx).

Since:

3.0

Method Summary

Modifier and Type	Method and Description
Sink<T>	build() Creates and returns the Sink with the components you supplied to this builder.
SinkBuilder<C,T>	destroyFn(ConsumerEx<? super C> destroyFn) Sets the function that will destroy the context object and perform any cleanup.
SinkBuilder<C,T>	flushFn(ConsumerEx<? super C> flushFn) Sets the function that implements the sink's flushing behavior.
SinkBuilder<C,T>	preferredLocalParallelism(int preferredLocalParallelism) Sets the local parallelism of the sink.
<T_NEW> SinkBuilder<C,T_NEW>	receiveFn(BiConsumerEx<? super C,? super T_NEW> receiveFn) Sets the function Jet will call upon receiving an item.
static <C> SinkBuilder<C,Void>	sinkBuilder(String name, FunctionEx<Processor.Context,? extends C> createFn) Returns a builder object that offers a step-by-step fluent API to build a custom Sink for the Pipeline API.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

sinkBuilder

@Nonnull
public static <C> SinkBuilder<C,Void> sinkBuilder(@Nonnull
                                                           String name,
                                                           @Nonnull
                                                           FunctionEx<Processor.Context,? extends C> createFn)

Returns a builder object that offers a step-by-step fluent API to build a custom Sink for the Pipeline API. It allows you to keep a single-threaded and stateful context object, it got from your createFn, in each instance of a Jet worker dedicated to driving the sink. Its primary intended purpose is to serve as the holder of references to external resources and optional buffers. Keep in mind that only the context object may be stateful; the functions you provide must hold no mutable state of their own.

These are the callback functions you can provide to implement the sink's behavior:

1. createFn creates the context object. Gets the processor context as argument which can be used to obtain local Jet instance, global processor index etc. It will be called once for each worker thread. This component is required.
2. onReceiveFn gets notified of each item the sink receives and (typically) passes it to the context. This component is required.
3. flushFn flushes the context. This component is optional.
4. destroyFn destroys the context. This component is optional.

The returned sink will be non-cooperative and will have preferred local parallelism of 1. It doesn't participate in the fault-tolerance protocol, which means you can't remember across a job restart which items you already received. The sink will still receive each item at least once, thus complying with the at-least-once processing guarantee. If the sink is idempotent (suppresses duplicate items), it will also be compatible with the exactly-once guarantee.

All the functions must be stateless.

Type Parameters:

C - type of the context object

Parameters:

name - the name of the processor

createFn - the function to create the sink context, given a processor context. It must be stateless.

Since:

3.0

receiveFn

@Nonnull
public <T_NEW> SinkBuilder<C,T_NEW> receiveFn(@Nonnull
                                                       BiConsumerEx<? super C,? super T_NEW> receiveFn)

Sets the function Jet will call upon receiving an item. The function receives two arguments: the context object (as provided by the createFn and the received item. Its job is to push the item to the context.

Type Parameters:

T_NEW - type of the items the sink will accept

Parameters:

receiveFn - the "add item to the context" function. It must be stateless.

flushFn

@Nonnull
public SinkBuilder<C,T> flushFn(@Nonnull
                                         ConsumerEx<? super C> flushFn)

Sets the function that implements the sink's flushing behavior. If your context object is buffered, instead of relying on some automatic flushing policy you can provide this function so Jet can choose the best moment to flush.

You are not required to provide this function in case your implementation doesn't need it.

Parameters:

flushFn - the optional "flush the context" function. It must be stateless.

destroyFn

@Nonnull
public SinkBuilder<C,T> destroyFn(@Nonnull
                                           ConsumerEx<? super C> destroyFn)

Sets the function that will destroy the context object and perform any cleanup. The function is called when the job has been completed or cancelled. Jet guarantees that no new items will be received in between the last call to flushFn and the call to destroyFn.

You are not required to provide this function in case your implementation doesn't need it.

Parameters:

destroyFn - the optional "destroy the context object" function. It must be stateless.

preferredLocalParallelism

@Nonnull
public SinkBuilder<C,T> preferredLocalParallelism(int preferredLocalParallelism)

Sets the local parallelism of the sink. On each member of the cluster Jet will create this many parallel processors for the sink. To identify each processor instance, your createFn can consult procContext.totalParallelism() and procContext.globalProcessorIndex(). Jet calls createFn exactly once with each globalProcessorIndex from 0 to totalParallelism - 1.

The default value of this property is 1.

build

@Nonnull
public Sink<T> build()

Creates and returns the Sink with the components you supplied to this builder.