Package com.hazelcast.jet.hadoop

Class HadoopProcessors

java.lang.Object

com.hazelcast.jet.hadoop.HadoopProcessors

public final class HadoopProcessors
extends Object

Static utility class with factories of Apache Hadoop Hadoop source and sink processors.

Since:

Jet 3.0

Method Summary

Modifier and Type	Method	Description
static <K, V, R> ProcessorMetaSupplier	readHadoopP(Permission permission, ConsumerEx<org.apache.hadoop.conf.Configuration> configureFn, BiFunctionEx<K,V,R> projectionFn)	Returns a supplier of processors for FileSources.files(String).
static <K, V, R> ProcessorMetaSupplier	readHadoopP(org.apache.hadoop.conf.Configuration configuration, BiFunctionEx<K,V,R> projectionFn)	Returns a supplier of processors for HadoopSources.inputFormat(Configuration, BiFunctionEx).
static <E, K, V> ProcessorMetaSupplier	writeHadoopP(org.apache.hadoop.conf.Configuration configuration, FunctionEx<? super E,K> extractKeyFn, FunctionEx<? super E,V> extractValueFn)	Returns a supplier of processors for HadoopSinks.outputFormat(Configuration, FunctionEx, FunctionEx).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

readHadoopP

@Nonnull
public static <K,
V,
R> ProcessorMetaSupplier readHadoopP(@Nonnull
 org.apache.hadoop.conf.Configuration configuration,
 @Nonnull
 BiFunctionEx<K,V,R> projectionFn)

Returns a supplier of processors for HadoopSources.inputFormat(Configuration, BiFunctionEx).

readHadoopP

@Nonnull
public static <K,
V,
R> ProcessorMetaSupplier readHadoopP(@Nullable
 Permission permission,
 @Nonnull
 ConsumerEx<org.apache.hadoop.conf.Configuration> configureFn,
 @Nonnull
 BiFunctionEx<K,V,R> projectionFn)

Returns a supplier of processors for FileSources.files(String). The configuration happens via provided configureFn function on the job coordinator node. This is useful in cases where setting up the configuration requires access to the server and only the cluster members have the access.

writeHadoopP

@Nonnull
public static <E,
K,
V> ProcessorMetaSupplier writeHadoopP(@Nonnull
 org.apache.hadoop.conf.Configuration configuration,
 @Nonnull
 FunctionEx<? super E,K> extractKeyFn,
 @Nonnull
 FunctionEx<? super E,V> extractValueFn)

Returns a supplier of processors for HadoopSinks.outputFormat(Configuration, FunctionEx, FunctionEx).