public final class HadoopSinks extends Object
Modifier and Type | Method and Description |
---|---|
static <K,V> Sink<Map.Entry<K,V>> |
outputFormat(org.apache.hadoop.conf.Configuration configuration)
Convenience for
outputFormat(Configuration, FunctionEx,
FunctionEx) which expects Map.Entry<K, V> as
input and extracts its key and value parts to be written to HDFS. |
static <E,K,V> Sink<E> |
outputFormat(org.apache.hadoop.conf.Configuration configuration,
FunctionEx<? super E,K> extractKeyF,
FunctionEx<? super E,V> extractValueF)
Returns a sink that writes to Apache Hadoop HDFS.
|
@Nonnull public static <E,K,V> Sink<E> outputFormat(@Nonnull org.apache.hadoop.conf.Configuration configuration, @Nonnull FunctionEx<? super E,K> extractKeyF, @Nonnull FunctionEx<? super E,V> extractValueF)
configuration
.
The sink creates a number of files in the output path, identified by the
cluster member UUID and the Processor
index. Unlike MapReduce,
the data in the files is not sorted by key.
The supplied Configuration
must specify an OutputFormat
class with a path.
The processor will use either the new or the old MapReduce API based on
the key which stores the OutputFormat
configuration. If it's
stored under }, the new API
will be used. Otherwise, the old API will be used. If you get the
configuration from JobContextImpl.getConfiguration()
, the new API will be
used.
No state is saved to snapshot for this sink. After the job is restarted, the files will be overwritten. If the cluster members change, some files will be overwritten and some not - we don't clean the directory before the execution starts.
The default local parallelism for this processor is 2 (or less if less CPUs are available).
E
- stream item typeK
- type of key to write to HDFSV
- type of value to write to HDFSconfiguration
- Configuration
used for output format configurationextractKeyF
- mapper to map a key to another keyextractValueF
- mapper to map a value to another value@Nonnull public static <K,V> Sink<Map.Entry<K,V>> outputFormat(@Nonnull org.apache.hadoop.conf.Configuration configuration)
outputFormat(Configuration, FunctionEx,
FunctionEx)
which expects Map.Entry<K, V>
as
input and extracts its key and value parts to be written to HDFS.Copyright © 2023 Hazelcast, Inc.. All rights reserved.