public final class Sinks extends Object
SinkStage and accepts no downstream stages.
The default local parallelism for the sinks in this class is typically 1, check the documentation of individual methods.
| Modifier and Type | Method and Description |
|---|---|
static <W,T> SinkBuilder<W,T> |
builder(DistributedFunction<? super JetInstance,? extends W> createFn)
Returns a builder object that offers a step-by-step fluent API to build a
custom sink.
|
static <T extends Map.Entry> |
cache(String cacheName)
Returns a sink that puts
Map.Entrys it receives into a Hazelcast
ICache with the specified name. |
static <T> Sink<T> |
files(String directoryName)
Convenience for
files(String, DistributedFunction, Charset,
boolean) with the UTF-8 charset and with overwriting of existing files. |
static <T> Sink<T> |
files(String directoryName,
DistributedFunction<T,String> toStringFn)
Convenience for
files(String, DistributedFunction, Charset,
boolean) with the UTF-8 charset and with overwriting of existing files. |
static <T> Sink<T> |
files(String directoryName,
DistributedFunction<T,String> toStringFn,
Charset charset,
boolean append)
Returns a sink that that writes the items it receives to files.
|
static <T> Sink<T> |
fromProcessor(String sinkName,
ProcessorMetaSupplier metaSupplier)
Returns a sink constructed directly from the given Core API processor
meta-supplier.
|
static <T> Sink<T> |
list(String listName)
Returns a sink that adds the items it receives to a Hazelcast
IList with the specified name. |
static <T> Sink<T> |
logger()
|
static <T> Sink<T> |
logger(DistributedFunction<T,String> toStringFn)
Returns a sink that logs all the data items it receives, at the INFO
level to the log category
WriteLoggerP. |
static <T extends Map.Entry> |
map(String mapName)
Returns a sink that puts
Map.Entrys it receives into a Hazelcast
IMap with the specified name. |
static <E,K,V> Sink<E> |
mapWithEntryProcessor(String mapName,
DistributedFunction<E,K> toKeyFn,
DistributedFunction<E,EntryProcessor<K,V>> toEntryProcessorFn)
Returns a sink that uses the items it receives to create
EntryProcessors it submits to a Hazelcast IMap with the
specified name. |
static <E extends Map.Entry,V> |
mapWithMerging(String mapName,
DistributedBinaryOperator<V> mergeFn)
|
static <E,K,V> Sink<E> |
mapWithMerging(String mapName,
DistributedFunction<E,K> toKeyFn,
DistributedFunction<E,V> toValueFn,
DistributedBinaryOperator<V> mergeFn)
Returns a sink that uses the supplied functions to extract the key
and value with which to update a Hazelcast
IMap. |
static <E extends Map.Entry,V> |
mapWithUpdating(String mapName,
DistributedBiFunction<V,E,V> updateFn)
Convenience for
mapWithUpdating(java.lang.String, com.hazelcast.jet.function.DistributedFunction<E, K>, com.hazelcast.jet.function.DistributedBiFunction<V, E, V>) with Map.Entry as
input item. |
static <E,K,V> Sink<E> |
mapWithUpdating(String mapName,
DistributedFunction<E,K> toKeyFn,
DistributedBiFunction<V,E,V> updateFn)
Returns a sink that uses the supplied key-extracting and value-updating
functions to update a Hazelcast
IMap. |
static <T extends Map.Entry> |
remoteCache(String cacheName,
ClientConfig clientConfig)
Returns a sink that puts
Map.Entrys it receives into a Hazelcast
ICache with the specified name in a remote cluster identified by
the supplied ClientConfig. |
static <T> Sink<T> |
remoteList(String listName,
ClientConfig clientConfig)
Returns a sink that adds the items it receives to a Hazelcast
IList with the specified name in a remote cluster identified by the
supplied ClientConfig. |
static <T extends Map.Entry> |
remoteMap(String mapName,
ClientConfig clientConfig)
Returns a sink that puts
Map.Entrys it receives into a Hazelcast
IMap with the specified name in a remote cluster identified by
the supplied ClientConfig. |
static <E,K,V> Sink<E> |
remoteMapWithEntryProcessor(String mapName,
ClientConfig clientConfig,
DistributedFunction<E,K> toKeyFn,
DistributedFunction<E,EntryProcessor<K,V>> toEntryProcessorFn)
Returns a sink equivalent to
mapWithEntryProcessor(java.lang.String, com.hazelcast.jet.function.DistributedFunction<E, K>, com.hazelcast.jet.function.DistributedFunction<E, com.hazelcast.map.EntryProcessor<K, V>>), but for a map
in a remote Hazelcast cluster identified by the supplied ClientConfig. |
static <E extends Map.Entry,V> |
remoteMapWithMerging(String mapName,
ClientConfig clientConfig,
DistributedBinaryOperator<V> mergeFn)
|
static <E,K,V> Sink<E> |
remoteMapWithMerging(String mapName,
ClientConfig clientConfig,
DistributedFunction<E,K> toKeyFn,
DistributedFunction<E,V> toValueFn,
DistributedBinaryOperator<V> mergeFn)
Returns a sink equivalent to
mapWithMerging(java.lang.String, com.hazelcast.jet.function.DistributedFunction<E, K>, com.hazelcast.jet.function.DistributedFunction<E, V>, com.hazelcast.jet.function.DistributedBinaryOperator<V>), but for a map
in a remote Hazelcast cluster identified by the supplied ClientConfig. |
static <E extends Map.Entry,V> |
remoteMapWithUpdating(String mapName,
ClientConfig clientConfig,
DistributedBiFunction<V,E,V> updateFn)
|
static <E,K,V> Sink<E> |
remoteMapWithUpdating(String mapName,
ClientConfig clientConfig,
DistributedFunction<E,K> toKeyFn,
DistributedBiFunction<V,E,V> updateFn)
Returns a sink equivalent to
mapWithUpdating(java.lang.String, com.hazelcast.jet.function.DistributedFunction<E, K>, com.hazelcast.jet.function.DistributedBiFunction<V, E, V>), but for a map
in a remote Hazelcast cluster identified by the supplied ClientConfig. |
static <T> Sink<T> |
socket(String host,
int port)
Convenience for
socket(String, int, DistributedFunction,
Charset) with Object.toString as the conversion function and
UTF-8 as the charset. |
static <T> Sink<T> |
socket(String host,
int port,
DistributedFunction<T,String> toStringFn)
Convenience for
socket(String, int, DistributedFunction,
Charset) with UTF-8 as the charset. |
static <T> Sink<T> |
socket(String host,
int port,
DistributedFunction<T,String> toStringFn,
Charset charset)
Returns a sink that connects to the specified TCP socket and writes to
it a string representation of the items it receives.
|
@Nonnull public static <T> Sink<T> fromProcessor(@Nonnull String sinkName, @Nonnull ProcessorMetaSupplier metaSupplier)
The default local parallelism for this source is specified inside the
metaSupplier.
sinkName - user-friendly sink namemetaSupplier - the processor meta-supplier@Nonnull public static <T extends Map.Entry> Sink<T> map(@Nonnull String mapName)
Map.Entrys it receives into a Hazelcast
IMap with the specified name.
This sink provides the exactly-once guarantee thanks to idempotent updates. It means that the value with the same key is not appended, but overwritten. After the job is restarted from snapshot, duplicate items will not change the state in the target map.
The default local parallelism for this sink is 1.
@Nonnull public static <T extends Map.Entry> Sink<T> remoteMap(@Nonnull String mapName, @Nonnull ClientConfig clientConfig)
Map.Entrys it receives into a Hazelcast
IMap with the specified name in a remote cluster identified by
the supplied ClientConfig.
This sink provides the exactly-once guarantee thanks to idempotent updates. It means that the value with the same key is not appended, but overwritten. After the job is restarted from snapshot, duplicate items will not change the state in the target map.
The default local parallelism for this sink is 1.
@Nonnull public static <E,K,V> Sink<E> mapWithMerging(@Nonnull String mapName, @Nonnull DistributedFunction<E,K> toKeyFn, @Nonnull DistributedFunction<E,V> toValueFn, @Nonnull DistributedBinaryOperator<V> mergeFn)
IMap. If the map
already contains the key, it applies the given mergeFn to
resolve the existing and the proposed value into the value to use. If
the value comes out as null, it removes the key from the map.
Expressed as code, the sink performs the equivalent of the following for
each item:
K key = toKeyFn.apply(item);
V oldValue = map.get(key);
V newValue = toValueFn.apply(item);
V resolved = (oldValue == null)
? newValue
: mergeFn.apply(oldValue, newValue);
if (value == null)
map.remove(key);
else
map.put(key, value);
This sink supports exactly-once processing only if the
supplied merge function performs idempotent updates, i.e.,
it satisfies the rule
mergeFn.apply(oldValue, toValueFn.apply(e)).equals(oldValue)
for any e that was already observed.
Note: This operation is NOT lock-aware, it will process the
entries no matter if they are locked or not. Use mapWithEntryProcessor(java.lang.String, com.hazelcast.jet.function.DistributedFunction<E, K>, com.hazelcast.jet.function.DistributedFunction<E, com.hazelcast.map.EntryProcessor<K, V>>)
if you need locking.
The default local parallelism for this sink is 1.
E - input item typeK - key typeV - value typemapName - name of the maptoKeyFn - function that extracts the key from the input itemtoValueFn - function that extracts the value from the input itemmergeFn - function that merges the existing value with the value acquired from the
received item@Nonnull public static <E,K,V> Sink<E> remoteMapWithMerging(@Nonnull String mapName, @Nonnull ClientConfig clientConfig, @Nonnull DistributedFunction<E,K> toKeyFn, @Nonnull DistributedFunction<E,V> toValueFn, @Nonnull DistributedBinaryOperator<V> mergeFn)
mapWithMerging(java.lang.String, com.hazelcast.jet.function.DistributedFunction<E, K>, com.hazelcast.jet.function.DistributedFunction<E, V>, com.hazelcast.jet.function.DistributedBinaryOperator<V>), but for a map
in a remote Hazelcast cluster identified by the supplied ClientConfig.@Nonnull public static <E extends Map.Entry,V> Sink<E> mapWithMerging(@Nonnull String mapName, @Nonnull DistributedBinaryOperator<V> mergeFn)
@Nonnull public static <E extends Map.Entry,V> Sink<E> remoteMapWithMerging(@Nonnull String mapName, @Nonnull ClientConfig clientConfig, @Nonnull DistributedBinaryOperator<V> mergeFn)
@Nonnull public static <E,K,V> Sink<E> mapWithUpdating(@Nonnull String mapName, @Nonnull DistributedFunction<E,K> toKeyFn, @Nonnull DistributedBiFunction<V,E,V> updateFn)
IMap. For each item it receives, it
applies toKeyFn to get the key and then applies updateFn to
the existing value in the map and the received item to acquire the new
value to associate with the key. If the new value is null, it
removes the key from the map. Expressed as code, the sink performs the
equivalent of the following for each item:
K key = toKeyFn.apply(item);
V oldValue = map.get(key);
V newValue = updateFn.apply(oldValue, item);
if (newValue == null)
map.remove(key);
else
map.put(key, newValue);
This sink supports exactly-once processing only if the
supplied update function performs idempotent updates, i.e., it
satisfies the rule updateFn.apply(v, e).equals(v) for any
e that was already observed.
Note: This operation is NOT lock-aware, it will process the entries
no matter if they are locked or not.
Use mapWithEntryProcessor(java.lang.String, com.hazelcast.jet.function.DistributedFunction<E, K>, com.hazelcast.jet.function.DistributedFunction<E, com.hazelcast.map.EntryProcessor<K, V>>) if you need locking.
The default local parallelism for this sink is 1.
E - input item typeK - key typeV - value typemapName - name of the maptoKeyFn - function that extracts the key from the input itemupdateFn - function that receives the existing map value and the item
and returns the new map value@Nonnull public static <E,K,V> Sink<E> remoteMapWithUpdating(@Nonnull String mapName, @Nonnull ClientConfig clientConfig, @Nonnull DistributedFunction<E,K> toKeyFn, @Nonnull DistributedBiFunction<V,E,V> updateFn)
mapWithUpdating(java.lang.String, com.hazelcast.jet.function.DistributedFunction<E, K>, com.hazelcast.jet.function.DistributedBiFunction<V, E, V>), but for a map
in a remote Hazelcast cluster identified by the supplied ClientConfig.@Nonnull public static <E extends Map.Entry,V> Sink<E> mapWithUpdating(@Nonnull String mapName, @Nonnull DistributedBiFunction<V,E,V> updateFn)
mapWithUpdating(java.lang.String, com.hazelcast.jet.function.DistributedFunction<E, K>, com.hazelcast.jet.function.DistributedBiFunction<V, E, V>) with Map.Entry as
input item.@Nonnull public static <E extends Map.Entry,V> Sink<E> remoteMapWithUpdating(@Nonnull String mapName, @Nonnull ClientConfig clientConfig, @Nonnull DistributedBiFunction<V,E,V> updateFn)
@Nonnull public static <E,K,V> Sink<E> mapWithEntryProcessor(@Nonnull String mapName, @Nonnull DistributedFunction<E,K> toKeyFn, @Nonnull DistributedFunction<E,EntryProcessor<K,V>> toEntryProcessorFn)
EntryProcessors it submits to a Hazelcast IMap with the
specified name. For each received item it applies toKeyFn to
get the key and toEntryProcessorFn to get the entry processor,
and then submits the key and the entry processor to the Hazelcast
cluster, which will internally apply the entry processor to the key.
As opposed to mapWithUpdating(java.lang.String, com.hazelcast.jet.function.DistributedFunction<E, K>, com.hazelcast.jet.function.DistributedBiFunction<V, E, V>) and mapWithMerging(java.lang.String, com.hazelcast.jet.function.DistributedFunction<E, K>, com.hazelcast.jet.function.DistributedFunction<E, V>, com.hazelcast.jet.function.DistributedBinaryOperator<V>),
this sink does not use batching and submits a separate entry processor
for each received item. For use cases that are efficiently solvable
using those sinks, this one will perform worse. It should be used only
when they are not applicable.
If your entry processors take a long time to update a value, consider
using entry processors that implement Offloadable. This will
avoid blocking the Hazelcast partition thread during large update
operations.
This sink supports exactly-once processing only if the supplied entry processor performs idempotent updates, i.e., the resulting value would be the same if an entry processor was run on the same entry more than once.
Note: Unlike mapWithUpdating(java.lang.String, com.hazelcast.jet.function.DistributedFunction<E, K>, com.hazelcast.jet.function.DistributedBiFunction<V, E, V>) and mapWithMerging(java.lang.String, com.hazelcast.jet.function.DistributedFunction<E, K>, com.hazelcast.jet.function.DistributedFunction<E, V>, com.hazelcast.jet.function.DistributedBinaryOperator<V>),
this operation is lock-aware. If the key is locked,
the EntryProcessor will wait until it acquires the lock.
The default local parallelism for this sink is 1.
E - input item typeK - key typeV - value typemapName - name of the maptoKeyFn - function that extracts the key from the input itemtoEntryProcessorFn - function that returns the EntryProcessor
to apply to the key@Nonnull public static <E,K,V> Sink<E> remoteMapWithEntryProcessor(@Nonnull String mapName, @Nonnull ClientConfig clientConfig, @Nonnull DistributedFunction<E,K> toKeyFn, @Nonnull DistributedFunction<E,EntryProcessor<K,V>> toEntryProcessorFn)
mapWithEntryProcessor(java.lang.String, com.hazelcast.jet.function.DistributedFunction<E, K>, com.hazelcast.jet.function.DistributedFunction<E, com.hazelcast.map.EntryProcessor<K, V>>), but for a map
in a remote Hazelcast cluster identified by the supplied ClientConfig.@Nonnull public static <T extends Map.Entry> Sink<T> cache(@Nonnull String cacheName)
Map.Entrys it receives into a Hazelcast
ICache with the specified name.
This sink provides the exactly-once guarantee thanks to idempotent updates. It means that the value with the same key is not appended, but overwritten. After the job is restarted from snapshot, duplicate items will not change the state in the target map.
The default local parallelism for this sink is 1.
@Nonnull public static <T extends Map.Entry> Sink<T> remoteCache(@Nonnull String cacheName, @Nonnull ClientConfig clientConfig)
Map.Entrys it receives into a Hazelcast
ICache with the specified name in a remote cluster identified by
the supplied ClientConfig.
This sink provides the exactly-once guarantee thanks to idempotent updates. It means that the value with the same key is not appended, but overwritten. After the job is restarted from snapshot, duplicate items will not change the state in the target map.
The default local parallelism for this sink is 1.
@Nonnull public static <T> Sink<T> list(@Nonnull String listName)
IList with the specified name.
No state is saved to snapshot for this sink. After the job is restarted, the items will likely be duplicated, providing an at-least-once guarantee.
The default local parallelism for this sink is 1.
@Nonnull public static <T> Sink<T> remoteList(@Nonnull String listName, @Nonnull ClientConfig clientConfig)
IList with the specified name in a remote cluster identified by the
supplied ClientConfig.
No state is saved to snapshot for this sink. After the job is restarted, the items will likely be duplicated, providing an at-least-once guarantee.
The default local parallelism for this sink is 1.
@Nonnull public static <T> Sink<T> socket(@Nonnull String host, int port, @Nonnull DistributedFunction<T,String> toStringFn, @Nonnull Charset charset)
toStringFn
function and encodes the string using the supplied Charset. It
follows each item with a newline character.
No state is saved to snapshot for this sink. After the job is restarted, the items will likely be duplicated, providing an at-least-once guarantee.
The default local parallelism for this sink is 1.
@Nonnull public static <T> Sink<T> socket(@Nonnull String host, int port, @Nonnull DistributedFunction<T,String> toStringFn)
socket(String, int, DistributedFunction,
Charset) with UTF-8 as the charset.@Nonnull public static <T> Sink<T> socket(@Nonnull String host, int port)
socket(String, int, DistributedFunction,
Charset) with Object.toString as the conversion function and
UTF-8 as the charset.@Nonnull public static <T> Sink<T> files(@Nonnull String directoryName, @Nonnull DistributedFunction<T,String> toStringFn, @Nonnull Charset charset, boolean append)
The sink converts an item to its string representation using the
supplied toStringFn function and encodes the string using the
supplied Charset. It follows each item with a platform-specific
line separator.
No state is saved to snapshot for this sink. After the job is restarted, the items will likely be duplicated, providing an at-least-once guarantee.
The default local parallelism for this sink is 1.
directoryName - directory to create the files in. Will be created
if it doesn't exist. Must be the same on all members.toStringFn - a function to convert items to String (a formatter)charset - charset used to encode the file outputappend - whether to append (true) or overwrite (false)
an existing file@Nonnull public static <T> Sink<T> files(@Nonnull String directoryName, @Nonnull DistributedFunction<T,String> toStringFn)
files(String, DistributedFunction, Charset,
boolean) with the UTF-8 charset and with overwriting of existing files.@Nonnull public static <T> Sink<T> files(@Nonnull String directoryName)
files(String, DistributedFunction, Charset,
boolean) with the UTF-8 charset and with overwriting of existing files.@Nonnull public static <T> Sink<T> logger(@Nonnull DistributedFunction<T,String> toStringFn)
WriteLoggerP. It also logs watermark items, but at FINE level.
The sink logs each item on whichever cluster member it happens to receive it. Its primary purpose is for development, when running Jet on a local machine.
The default local parallelism for this sink is 1.
T - stream item typetoStringFn - a function that returns a string representation of a stream item@Nonnull public static <W,T> SinkBuilder<W,T> builder(@Nonnull DistributedFunction<? super JetInstance,? extends W> createFn)
W - type of the writer objectT - type of the items the sink will acceptcreateFn - function that creates the internal writer objectCopyright © 2018 Hazelcast, Inc.. All rights reserved.