Package com.hazelcast.jet.hadoop

Class HadoopProcessors

java.lang.Object

com.hazelcast.jet.hadoop.HadoopProcessors

public final class HadoopProcessors
extends java.lang.Object

Static utility class with factories of Apache Hadoop Hadoop source and sink processors.

Since:

Jet 3.0

Method Summary

Modifier and Type	Method	Description
static <K,V,R> ProcessorMetaSupplier	readHadoopP(java.security.Permission permission, ConsumerEx<org.apache.hadoop.conf.Configuration> configureFn, BiFunctionEx<K,V,R> projectionFn)	Returns a supplier of processors for FileSources.files(String).
static <K,V,R> ProcessorMetaSupplier	readHadoopP(org.apache.hadoop.conf.Configuration configuration, BiFunctionEx<K,V,R> projectionFn)	Returns a supplier of processors for HadoopSources.inputFormat(Configuration, BiFunctionEx).
static <E,K,V> ProcessorMetaSupplier	writeHadoopP(org.apache.hadoop.conf.Configuration configuration, FunctionEx<? super E,K> extractKeyFn, FunctionEx<? super E,V> extractValueFn)	Returns a supplier of processors for HadoopSinks.outputFormat(Configuration, FunctionEx, FunctionEx).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

readHadoopP

@Nonnull
public static <K,V,R> ProcessorMetaSupplier readHadoopP(@Nonnull
                                                                    org.apache.hadoop.conf.Configuration configuration,
                                                                    @Nonnull
                                                                    BiFunctionEx<K,V,R> projectionFn)

Returns a supplier of processors for HadoopSources.inputFormat(Configuration, BiFunctionEx).

readHadoopP

@Nonnull
public static <K,V,R> ProcessorMetaSupplier readHadoopP(@Nullable
                                                                    java.security.Permission permission,
                                                                    @Nonnull
                                                                    ConsumerEx<org.apache.hadoop.conf.Configuration> configureFn,
                                                                    @Nonnull
                                                                    BiFunctionEx<K,V,R> projectionFn)

Returns a supplier of processors for FileSources.files(String). The configuration happens via provided configureFn function on the job coordinator node. This is useful in cases where setting up the configuration requires access to the server and only the cluster members have the access.

writeHadoopP

@Nonnull
public static <E,K,V> ProcessorMetaSupplier writeHadoopP(@Nonnull
                                                                     org.apache.hadoop.conf.Configuration configuration,
                                                                     @Nonnull
                                                                     FunctionEx<? super E,K> extractKeyFn,
                                                                     @Nonnull
                                                                     FunctionEx<? super E,V> extractValueFn)

Returns a supplier of processors for HadoopSinks.outputFormat(Configuration, FunctionEx, FunctionEx).