Package com.hazelcast.jet.pipeline

Class SinkBuilder<C,T>

java.lang.Object
com.hazelcast.jet.pipeline.SinkBuilder<C,T>
Type Parameters:
C - type of the context object
T - type of the items the sink will accept
public final class SinkBuilder<C,T> extends Object
See sinkBuilder(String, FunctionEx).
Since:
Jet 3.0

  • Method Details

    • sinkBuilder

      @Nonnull public static <C> SinkBuilder<C,Void> sinkBuilder(@Nonnull String name, @Nonnull FunctionEx<Processor.Context,? extends C> createFn)
      Returns a builder object that offers a step-by-step fluent API to build a custom Sink for the Pipeline API. It allows you to keep a single-threaded and stateful context object, it got from your createFn, in each instance of a Jet worker dedicated to driving the sink. Its primary intended purpose is to serve as the holder of references to external resources and optional buffers. Keep in mind that only the context object may be stateful; the functions you provide must hold no mutable state of their own.

      These are the callback functions you can provide to implement the sink's behavior:

      1. createFn creates the context object. Gets the processor context as argument which can be used to obtain local Jet instance, global processor index etc. It will be called once for each worker thread. This component is required.
      2. onReceiveFn gets notified of each item the sink receives and (typically) passes it to the context. This component is required.
      3. flushFn flushes the context. This component is optional.
      4. destroyFn destroys the context. This component is optional.
      The returned sink will be non-cooperative and will have preferred local parallelism of 1. It doesn't participate in the fault-tolerance protocol, which means you can't remember across a job restart which items you already received. The sink will still receive each item at least once, thus complying with the at-least-once processing guarantee. If the sink is idempotent (suppresses duplicate items), it will also be compatible with the exactly-once guarantee.

      All the functions must be stateless.

      Type Parameters:
      C - type of the context object
      Parameters:
      name - the name of the processor
      createFn - the function to create the sink context, given a processor context. It must be stateless.
      Since:
      Jet 3.0

    • receiveFn

      @Nonnull public <T_NEW> SinkBuilder<C,T_NEW> receiveFn(@Nonnull BiConsumerEx<? super C,? super T_NEW> receiveFn)
      Sets the function Jet will call upon receiving an item. The function receives two arguments: the context object (as provided by the createFn) and the received item. Its job is to push the item to the context.
      Type Parameters:
      T_NEW - type of the items the sink will accept
      Parameters:
      receiveFn - the "add item to the context" function. It must be stateless.

    • flushFn

      @Nonnull public SinkBuilder<C,T> flushFn(@Nonnull ConsumerEx<? super C> flushFn)
      Sets the function that implements the sink's flushing behavior. If your context object is buffered, instead of relying on some automatic flushing policy you can provide this function so Jet can choose the best moment to flush.

      You are not required to provide this function in case your implementation doesn't need it.

      Parameters:
      flushFn - the optional "flush the context" function. It must be stateless.

    • destroyFn

      @Nonnull public SinkBuilder<C,T> destroyFn(@Nonnull ConsumerEx<? super C> destroyFn)
      Sets the function that will destroy the context object and perform any cleanup. The function is called when the job has been completed or cancelled. Jet guarantees that no new items will be received in between the last call to flushFn and the call to destroyFn.

      You are not required to provide this function in case your implementation doesn't need it.

      Parameters:
      destroyFn - the optional "destroy the context object" function. It must be stateless.

    • permission

      public SinkBuilder<C,T> permission(@Nonnull Permission permission)
      Sets the permission required to use this sink when the security is enabled. The default value is null which means there is no restriction to use this sink. Security is an enterprise feature.
      Parameters:
      permission - the required permission to use this sink when security is enabled.

    • preferredLocalParallelism

      @Nonnull public SinkBuilder<C,T> preferredLocalParallelism(int preferredLocalParallelism)
      Sets the local parallelism of the sink. On each member of the cluster Jet will create this many parallel processors for the sink. To identify each processor instance, your createFn can consult procContext.totalParallelism() and procContext.globalProcessorIndex(). Jet calls createFn exactly once with each globalProcessorIndex from 0 to totalParallelism - 1.

      The default value of this property is 1.

    • build

      @Nonnull public Sink<T> build()
      Creates and returns the Sink with the components you supplied to this builder.