com.hazelcast.jet.hadoop

Class HadoopProcessors

java.lang.Object

com.hazelcast.jet.hadoop.HadoopProcessors

public final class HadoopProcessors
extends Object

Static utility class with factories of Apache Hadoop Hadoop source and sink processors.

Since:

Jet 3.0

Method Summary

Modifier and Type	Method and Description
static <K,V,R> ProcessorMetaSupplier	readHadoopP(org.apache.hadoop.conf.Configuration configuration, BiFunctionEx<K,V,R> projectionFn) Returns a supplier of processors for HadoopSources.inputFormat(Configuration, BiFunctionEx).
static <K,V,R> ProcessorMetaSupplier	readHadoopP(Permission permission, ConsumerEx<org.apache.hadoop.conf.Configuration> configureFn, BiFunctionEx<K,V,R> projectionFn) Returns a supplier of processors for FileSources.files(String).
static <E,K,V> ProcessorMetaSupplier	writeHadoopP(org.apache.hadoop.conf.Configuration configuration, FunctionEx<? super E,K> extractKeyFn, FunctionEx<? super E,V> extractValueFn) Returns a supplier of processors for HadoopSinks.outputFormat(Configuration, FunctionEx, FunctionEx).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

readHadoopP

@Nonnull
public static <K,V,R> ProcessorMetaSupplier readHadoopP(@Nonnull
                                                                 org.apache.hadoop.conf.Configuration configuration,
                                                                 @Nonnull
                                                                 BiFunctionEx<K,V,R> projectionFn)

Returns a supplier of processors for HadoopSources.inputFormat(Configuration, BiFunctionEx).

readHadoopP

@Nonnull
public static <K,V,R> ProcessorMetaSupplier readHadoopP(@Nullable
                                                                 Permission permission,
                                                                 @Nonnull
                                                                 ConsumerEx<org.apache.hadoop.conf.Configuration> configureFn,
                                                                 @Nonnull
                                                                 BiFunctionEx<K,V,R> projectionFn)

Returns a supplier of processors for FileSources.files(String). The configuration happens via provided configureFn function on the job coordinator node. This is useful in cases where setting up the configuration requires access to the server and only the cluster members have the access.

writeHadoopP

@Nonnull
public static <E,K,V> ProcessorMetaSupplier writeHadoopP(@Nonnull
                                                                  org.apache.hadoop.conf.Configuration configuration,
                                                                  @Nonnull
                                                                  FunctionEx<? super E,K> extractKeyFn,
                                                                  @Nonnull
                                                                  FunctionEx<? super E,V> extractValueFn)

Returns a supplier of processors for HadoopSinks.outputFormat(Configuration, FunctionEx, FunctionEx).