Class PythonTransforms


  • public final class PythonTransforms
    extends java.lang.Object
    Transforms which allow the user to call Python user-defined functions from inside a Jet pipeline.
    Since:
    Jet 4.0
    • Method Detail

      • mapUsingPython

        @Nonnull
        public static FunctionEx<StreamStage<java.lang.String>,​StreamStage<java.lang.String>> mapUsingPython​(@Nonnull
                                                                                                                   PythonServiceConfig cfg)
        A stage-transforming method that adds a "map using Python" pipeline stage. Use it with stage.apply(PythonService.mapUsingPython(pyConfig)). See PythonServiceConfig for more details.
      • mapUsingPython

        @Deprecated
        @Nonnull
        public static <K> FunctionEx<StreamStage<java.lang.String>,​StreamStage<java.lang.String>> mapUsingPython​(@Nonnull
                                                                                                                       FunctionEx<? super java.lang.String,​? extends K> keyFn,
                                                                                                                       @Nonnull
                                                                                                                       PythonServiceConfig cfg)
        Deprecated.
        Jet now has first-class support for data rebalancing, see GeneralStage.rebalance() and GeneralStage.rebalance(FunctionEx).
        A stage-transforming method that adds a partitioned "map using Python" pipeline stage. It applies partitioning using the supplied keyFn. You need partitioning if your input stream comes from a non-distributed data source (all data coming in on a single cluster member), in order to distribute the Python work across the whole cluster.

        Use it like this: stage.apply(PythonService.mapUsingPython(keyFn, pyConfig)). See PythonServiceConfig for more details.

      • mapUsingPythonBatch

        @Nonnull
        public static FunctionEx<BatchStage<java.lang.String>,​BatchStage<java.lang.String>> mapUsingPythonBatch​(@Nonnull
                                                                                                                      PythonServiceConfig cfg)
        A stage-transforming method that adds a "map using Python" pipeline stage. Use it with stage.apply(PythonService.mapUsingPythonBatch(pyConfig)). See PythonServiceConfig for more details.
      • mapUsingPythonBatch

        @Nonnull
        public static <K> FunctionEx<BatchStage<java.lang.String>,​BatchStage<java.lang.String>> mapUsingPythonBatch​(@Nonnull
                                                                                                                          FunctionEx<? super java.lang.String,​? extends K> keyFn,
                                                                                                                          @Nonnull
                                                                                                                          PythonServiceConfig cfg)
        A stage-transforming method that adds a partitioned "map using Python" pipeline stage. It applies partitioning using the supplied keyFn. You need partitioning if your input stream comes from a non-distributed data source (all data coming in on a single cluster member), in order to distribute the Python work across the whole cluster.

        Use it like this: stage.apply(PythonService.mapUsingPythonBatch(keyFn, pyConfig)). See PythonServiceConfig for more details.