Package com.hazelcast.jet.core

Interface ProcessorMetaSupplier

  • All Superinterfaces:
    java.io.Serializable
    All Known Implementing Classes:
    ProcessorMetaSupplier.RandomMemberPms, ProcessorMetaSupplier.SpecificMemberPms
    Functional Interface:
    This is a functional interface and can therefore be used as the assignment target for a lambda expression or method reference.
    @FunctionalInterface
public interface ProcessorMetaSupplier
extends java.io.Serializable
    Factory of ProcessorSupplier instances. The starting point of the chain leading to the eventual creation of Processor instances on each cluster member:
    1. client or member creates ProcessorMetaSupplier as a part of the DAG;
    2. client or member sends it to the job coordinator (if the member is job coordinator and the DAG consists of reusable ProcessorMetaSuppliers, the serialization may be skipped);
    3. the job coordinator uses it to create one ProcessorSupplier for each cluster member;
    4. serializes each ProcessorSupplier and sends it to its target member;
    5. the target member deserializes and uses it to instantiate as many instances of Processor as requested by the parallelism property on the corresponding Vertex.
    Before being asked to create ProcessorSuppliers this meta-supplier will be given access to the Hazelcast instance and, in particular, its cluster topology and partitioning services. It can use the information from these services to precisely parameterize each Processor instance that will be created on each member.
    Since:
    Jet 3.0

    • Method Detail

      • getRequiredPermission

        @Nullable
default java.security.Permission getRequiredPermission()
        Returns the required permission to execute the vertex which has this ProcessorMetaSupplier. This is an Enterprise feature.

      • getTags

        @Nonnull
default java.util.Map<java.lang.String,​java.lang.String> getTags()
        Returns the metadata on this supplier, a string-to-string map. There is no predefined metadata; this facility exists to allow the DAG vertices to contribute some information to the execution planning phase.
        Since:
        Jet 4.0

      • preferredLocalParallelism

        default int preferredLocalParallelism()
        Returns the local parallelism the vertex should be configured with. The default implementation returns Vertex.LOCAL_PARALLELISM_USE_DEFAULT.

      • init

        default void init​(@Nonnull
                  ProcessorMetaSupplier.Context context)
           throws java.lang.Exception
        Called on the cluster member that receives the job request. Gives access to the Hazelcast instance's services and provides the parallelism parameters determined from the cluster size.
        Throws:
        java.lang.Exception
        See Also:
        isReusable()

      • initIsCooperative

        default boolean initIsCooperative()
        Returns true if both the init(Context) and get(List) methods of this instance are cooperative. If they are not, the call to the init() and get() method is off-loaded to another thread.
        Since:
        5.2

      • get

        @Nonnull
java.util.function.Function<? super Address,​? extends ProcessorSupplier> get​(@Nonnull
                                                                                   java.util.List<Address> addresses)
        Called to create a mapping from member Address to the ProcessorSupplier that will be sent to that member. Jet calls this method with a list of all cluster members' addresses and the returned function must be a mapping that returns a non-null value for each given address.

        The method will be called once per job execution on the job's coordinator member. init() will have already been called.

        See Also:
        isReusable()

      • closeIsCooperative

        default boolean closeIsCooperative()
        Returns true if the close(Throwable) method of this instance is cooperative. If it's not, the call to the close() method is off-loaded to another thread.
        Since:
        5.2

      • close

        default void close​(@Nullable
                   java.lang.Throwable error)
            throws java.lang.Exception
        Called on coordinator member after execution has finished on all members, successfully or not. This method will be called after ProcessorSupplier.close(Throwable) has been called on all available members. The job can be restarted later.

        If there is an exception during the creation of the execution plan, this method will be called regardless of whether the init() or get() method have been called or not. If this method throws an exception, it will be logged and ignored; it won't be reported as a job failure.

        If you rely on the fact that this method is run once per cluster, it can happen that it is not called at all, if the coordinator member crashed. It can be also called multiple times, if the job restarts.

        Parameters:
        error - the exception (if any) that caused the job to fail; null in the case of successful job completion. Note that it might not be the actual error that caused the job to fail - it can be several other exceptions. We only guarantee that it's non-null if the job didn't complete successfully.
        Throws:
        java.lang.Exception
        See Also:
        isReusable()

      • of

        @Nonnull
static ProcessorMetaSupplier of​(int preferredLocalParallelism,
                                @Nullable
                                java.security.Permission permission,
                                @Nonnull
                                ProcessorSupplier procSupplier)
        Factory method that wraps the given ProcessorSupplier and returns the same instance for each given Address.
        Parameters:
        preferredLocalParallelism - the value to return from preferredLocalParallelism()
        permission - the required permission to run the processor
        procSupplier - the processor supplier

      • of

        @Nonnull
static ProcessorMetaSupplier of​(int preferredLocalParallelism,
                                @Nonnull
                                SupplierEx<? extends Processor> procSupplier)
        Factory method that wraps the given Supplier<Processor> and uses it as the supplier of all Processor instances. Specifically, returns a meta-supplier that will always return the result of calling ProcessorSupplier.of(SupplierEx).
        Parameters:
        preferredLocalParallelism - the value to return from preferredLocalParallelism()
        procSupplier - the supplier of processors

      • forceTotalParallelismOne

        @Nonnull
static ProcessorMetaSupplier forceTotalParallelismOne​(@Nonnull
                                                      ProcessorSupplier supplier,
                                                      @Nonnull
                                                      java.lang.String partitionKey,
                                                      @Nullable
                                                      java.security.Permission permission)
        Wraps the provided ProcessorSupplier into a meta-supplier that will only use the given ProcessorSupplier on a single node. The node will be chosen according to the partitionKey supplied. This is mainly provided as a convenience for implementing non-distributed sources where data can't be read in parallel by multiple consumers. When used as a sink or intermediate vertex, the DAG should ensure that only the processor instance on the designated node receives any data, otherwise an IllegalStateException will be thrown.

        The vertex containing the ProcessorMetaSupplier must have a local parallelism setting of 1, otherwise {code IllegalArgumentException} is thrown.

        Parameters:
        supplier - the supplier that will be wrapped
        partitionKey - the supplier will only be created on the node that owns the supplied partition key
        permission - the required permission to run the processor
        Returns:
        the wrapped ProcessorMetaSupplier
        Throws:
        java.lang.IllegalArgumentException - if vertex has local parallelism setting of greater than 1

      • forceTotalParallelismOne

        @Nonnull
static ProcessorMetaSupplier forceTotalParallelismOne​(@Nonnull
                                                      ProcessorSupplier supplier,
                                                      @Nonnull
                                                      Address memberAddress)
        Wraps the provided ProcessorSupplier into a meta-supplier that will only use the given ProcessorSupplier on a node with the given Address. This is mainly provided as a convenience for implementing non-distributed sources where data can't be read in parallel by multiple consumers. When used as a sink or intermediate vertex, the DAG should ensure that only the processor instance on the designated node receives any data, otherwise an IllegalStateException will be thrown.

        The vertex containing the ProcessorMetaSupplier must have a local parallelism setting of 1, otherwise {code IllegalArgumentException} is thrown.

        Parameters:
        supplier - the supplier that will be wrapped
        memberAddress - the supplier will only be created on the node with given Address
        Returns:
        the wrapped ProcessorMetaSupplier
        Throws:
        java.lang.IllegalArgumentException - if vertex has local parallelism setting of greater than 1

      • randomMember

        @Nonnull
@Beta
static ProcessorMetaSupplier randomMember​(@Nonnull
                                          ProcessorSupplier supplier)
        Wraps the provided ProcessorSupplier into a meta-supplier that will only use the given ProcessorSupplier on a random node
        Parameters:
        supplier - the supplier that will be wrapped
        Returns:
        the wrapped ProcessorMetaSupplier
        Since:
        5.3