Package com.hazelcast.jet.pipeline

Class SourceBuilder.Stream<T>

    • Method Detail

      • fillBufferFn

        @Nonnull
public <T_NEW> SourceBuilder.Stream<T_NEW> fillBufferFn​(@Nonnull
                                                        BiConsumerEx<? super C,​? super SourceBuilder.SourceBuffer<T_NEW>> fillBufferFn)
        Sets the function that Jet will call whenever it needs more data from your source. The function receives the context object obtained from createFn and Jet's buffer object. It should add some items to the buffer, ideally those it can produce without making any blocking calls. On any given invocation the function may also choose not to add any items. Jet will automatically employ an exponential backoff strategy to avoid calling your function in a tight loop, if the previous call didn't add any items to the buffer.

        The given SourceBuilder.SourceBuffer isn't thread-safe, you shouldn't pass it to other threads. For example, you shouldn't add to it in a callback of an asynchronous operation.

        Type Parameters:
        T_NEW - type of the emitted items
        Parameters:
        fillBufferFn - function that fills the buffer with source data. It must be stateless.
        Returns:
        this builder with the item type reset to the one inferred from fillBufferFn

      • destroyFn

        @Nonnull
public SourceBuilder.Stream<T> destroyFn​(@Nonnull
                                         ConsumerEx<? super C> pDestroyFn)
        Sets the function that Jet will call when it is done cleaning up after an execution. It gives you the opportunity to release any resources that your context object may be holding. Jet also calls this function when the user cancels or restarts the job.

        The function must be stateless.

      • distributed

        @Nonnull
public SourceBuilder.Stream<T> distributed​(int preferredLocalParallelism)
        Declares that you're creating a distributed source. On each member of the cluster Jet will create as many processors as you specify with the preferredLocalParallelism parameter. If you call this, you must ensure that all the source processors are coordinated and not emitting duplicated data. The createFn can consult processorContext.totalParallelism() and Processor.Context.globalProcessorIndex(). Jet calls createFn exactly once with each globalProcessorIndex from 0 to totalParallelism - 1 and you can use this to make all the instances agree on which part of the data to emit.

        If you don't call this method, there will be only one processor instance running on an arbitrary member.

        Parameters:
        preferredLocalParallelism - the requested number of processors on each cluster member

      • permission

        public SourceBuilder.Stream<T> permission​(@Nonnull
                                          java.security.Permission permission)
        Sets the the permission required to use this sink when the security is enabled. The default value is null which means there is no restriction to use this sink. Security is an enterprise feature.
        Parameters:
        permission - the required permission to use this sink when security is enabled.

      • build

        @Nonnull
public StreamSource<T> build()
        Builds and returns the unbounded stream source.