Package com.hazelcast.jet.hadoop

Class HadoopSinks

java.lang.Object
com.hazelcast.jet.hadoop.HadoopSinks
public final class HadoopSinks extends Object
Factories of Apache Hadoop sinks.
Since:
Jet 3.0

  • Method Details

    • outputFormat

      @Nonnull public static <E, K, V> Sink<E> outputFormat(@Nonnull org.apache.hadoop.conf.Configuration configuration, @Nonnull FunctionEx<? super E,K> extractKeyF, @Nonnull FunctionEx<? super E,V> extractValueF)
      Returns a sink that writes to Apache Hadoop HDFS. It transforms each received item to a key-value pair using the two supplied mapping functions. The type of the key and the value must conform to the expectations of the output format specified in the configuration.

      The sink creates a number of files in the output path, identified by the cluster member UUID and the Processor index. Unlike MapReduce, the data in the files is not sorted by key.

      The supplied Configuration must specify an OutputFormat class with a path.

      The processor will use either the new or the old MapReduce API based on the key which stores the OutputFormat configuration. If it's stored under "mapreduce.job.outputformat.class"}, the new API will be used. Otherwise, the old API will be used. If you get the configuration from JobContextImpl.getConfiguration(), the new API will be used.

      No state is saved to snapshot for this sink. After the job is restarted, the files will be overwritten. If the cluster members change, some files will be overwritten and some not - we don't clean the directory before the execution starts.

      The default local parallelism for this processor is 2 (or less if less CPUs are available).

      Type Parameters:
      E - stream item type
      K - type of key to write to HDFS
      V - type of value to write to HDFS
      Parameters:
      configuration - Configuration used for output format configuration
      extractKeyF - mapper to map a key to another key
      extractValueF - mapper to map a value to another value

    • outputFormat

      @Nonnull public static <K, V> Sink<Map.Entry<K,V>> outputFormat(@Nonnull org.apache.hadoop.conf.Configuration configuration)
      Convenience for outputFormat(Configuration, FunctionEx, FunctionEx) which expects Map.Entry<K, V> as input and extracts its key and value parts to be written to HDFS.