Uses of Interface
com.hazelcast.jet.pipeline.Sink
Package
Description
Apache Avro file read/write support for Hazelcast Jet.
Contains source/sink connectors that deal with Change Data Capture (CDC)
events from various databases as well as a generic connector for Debezium
CDC sources.
Contains sources and sinks for Elasticsearch 7
Apache Hadoop read/write support for Hazelcast Jet.
Apache Kafka reader/writer support for Hazelcast Jet.
Amazon Kinesis Data Streams producer/consumer support for Hazelcast Jet.
Contains sources and sinks for MongoDB.
The Pipeline API is Jet's high-level API to build and execute
distributed computation jobs.
This package contains various mock sources to help with pipeline testing
and development.
AWS S3 read/write support for Hazelcast Jet.
-
Uses of Sink in com.hazelcast.jet.avro
Modifier and TypeMethodDescriptionstatic <R> Sink<R>
Convenience forAvroSinks.files(String, Schema, SupplierEx)
which uses eitherSpecificDatumWriter
orReflectDatumWriter
depending on the suppliedrecordClass
.static Sink<org.apache.avro.generic.IndexedRecord>
Convenience forAvroSinks.files(String, Schema, SupplierEx)
which usesGenericDatumWriter
.static <R> Sink<R>
AvroSinks.files
(String directoryName, org.apache.avro.Schema schema, SupplierEx<org.apache.avro.io.DatumWriter<R>> datumWriterSupplier) Returns a sink that that writes the items it receives to Apache Avro files. -
Uses of Sink in com.hazelcast.jet.cdc
Modifier and TypeMethodDescriptionstatic <K,
V> Sink<ChangeRecord> CdcSinks.map
(IMap<? super K, ? super V> map, FunctionEx<? super ChangeRecord, ? extends K> keyFn, FunctionEx<? super ChangeRecord, ? extends V> valueFn) Returns a sink that applies the changes described by a Change Data Capture (CDC) stream to anIMap
.static <K,
V> Sink<ChangeRecord> CdcSinks.map
(String mapName, FunctionEx<? super ChangeRecord, ? extends K> keyFn, FunctionEx<? super ChangeRecord, ? extends V> valueFn) Returns a sink that applies the changes described by a Change Data Capture (CDC) stream to anIMap
.static <K,
V> Sink<ChangeRecord> CdcSinks.remoteMap
(String mapName, ClientConfig clientConfig, FunctionEx<? super ChangeRecord, ? extends K> keyFn, FunctionEx<? super ChangeRecord, ? extends V> valueFn) Returns a sink equivalent toCdcSinks.map(java.lang.String, com.hazelcast.function.FunctionEx<? super com.hazelcast.jet.cdc.ChangeRecord, ? extends K>, com.hazelcast.function.FunctionEx<? super com.hazelcast.jet.cdc.ChangeRecord, ? extends V>)
, but for a map in a remote Hazelcast cluster identified by the suppliedClientConfig
. -
Uses of Sink in com.hazelcast.jet.elastic
Modifier and TypeMethodDescriptionElasticSinkBuilder.build()
Create a sink that writes data into Elasticsearch based on this builder configurationstatic <T> Sink<T>
ElasticSinks.elastic
(FunctionEx<? super T, ? extends org.elasticsearch.action.DocWriteRequest<?>> mapToRequestFn) Creates an Elasticsearch sink, uses a local instance of Elasticsearchstatic <T> Sink<T>
ElasticSinks.elastic
(SupplierEx<org.elasticsearch.client.RestClientBuilder> clientFn, FunctionEx<? super T, ? extends org.elasticsearch.action.DocWriteRequest<?>> mapToRequestFn) Creates an Elasticsearch sink, uses a client obtained from clientFn and maps items using given mapToRequestFn -
Uses of Sink in com.hazelcast.jet.hadoop
Modifier and TypeMethodDescriptionHadoopSinks.outputFormat
(org.apache.hadoop.conf.Configuration configuration) Convenience forHadoopSinks.outputFormat(Configuration, FunctionEx, FunctionEx)
which expectsMap.Entry<K, V>
as input and extracts its key and value parts to be written to HDFS.static <E,
K, V> Sink<E> HadoopSinks.outputFormat
(org.apache.hadoop.conf.Configuration configuration, FunctionEx<? super E, K> extractKeyF, FunctionEx<? super E, V> extractValueF) Returns a sink that writes to Apache Hadoop HDFS. -
Uses of Sink in com.hazelcast.jet.kafka
Modifier and TypeMethodDescriptionKafkaSinks.Builder.build()
Builds the Sink object that you pass to theGeneralStage.writeTo(Sink)
method.static <E,
K, V> Sink<E> KafkaSinks.kafka
(DataConnectionRef dataConnectionRef, FunctionEx<? super E, org.apache.kafka.clients.producer.ProducerRecord<K, V>> toRecordFn) Returns a sink that publishes messages to Apache Kafka topic(s).KafkaSinks.kafka
(DataConnectionRef dataConnectionRef, String topic) Convenience forKafkaSinks.kafka(DataConnectionRef, String, FunctionEx, FunctionEx)
which expectsMap.Entry<K, V>
as input and extracts its key and value parts to be published to Kafka.static <E,
K, V> Sink<E> KafkaSinks.kafka
(DataConnectionRef dataConnectionRef, String topic, FunctionEx<? super E, K> extractKeyFn, FunctionEx<? super E, V> extractValueFn) Convenience forKafkaSinks.kafka(Properties, FunctionEx)
which creates aProducerRecord
using the given topic and the given key and value mapping functionsstatic <E,
K, V> Sink<E> KafkaSinks.kafka
(DataConnectionRef dataConnectionRef, Properties properties, String topic, FunctionEx<? super E, K> extractKeyFn, FunctionEx<? super E, V> extractValueFn) Convenience forKafkaSinks.kafka(Properties, FunctionEx)
which creates aProducerRecord
using the given topic and the given key and value mapping functions with additional properties availablestatic <E,
K, V> Sink<E> KafkaSinks.kafka
(Properties properties, FunctionEx<? super E, org.apache.kafka.clients.producer.ProducerRecord<K, V>> toRecordFn) Returns a sink that publishes messages to Apache Kafka topic(s).KafkaSinks.kafka
(Properties properties, String topic) Convenience forKafkaSinks.kafka(Properties, String, FunctionEx, FunctionEx)
which expectsMap.Entry<K, V>
as input and extracts its key and value parts to be published to Kafka.static <E,
K, V> Sink<E> KafkaSinks.kafka
(Properties properties, String topic, FunctionEx<? super E, K> extractKeyFn, FunctionEx<? super E, V> extractValueFn) Convenience forKafkaSinks.kafka(Properties, FunctionEx)
which creates aProducerRecord
using the given topic and the given key and value mapping functions -
Uses of Sink in com.hazelcast.jet.kinesis
-
Uses of Sink in com.hazelcast.jet.mongodb
Modifier and TypeMethodDescriptionMongoSinkBuilder.build()
Creates and returns the MongoDBSink
with the components you supplied to this builder.static Sink<org.bson.Document>
MongoSinks.mongodb
(DataConnectionRef dataConnectionRef, String database, String collection) static Sink<org.bson.Document>
-
Uses of Sink in com.hazelcast.jet.pipeline
Modifier and TypeMethodDescriptionFileSinkBuilder.build()
Creates and returns the fileSink
with the supplied components.JdbcSinkBuilder.build()
Creates and returns the JDBCSink
with the supplied components.JmsSinkBuilder.build()
Creates and returns the JMSSink
with the supplied components.MapSinkBuilder.build()
Build the sink.MapSinkEntryProcessorBuilder.build()
SinkBuilder.build()
Creates and returns theSink
with the components you supplied to this builder.Returns a sink that putsMap.Entry
s it receives into a HazelcastICache
with the specified name.static <T> Sink<T>
Convenience forSinks.filesBuilder(java.lang.String)
with the UTF-8 charset and with overwriting of existing files.static <T> Sink<T>
Sinks.fromProcessor
(String sinkName, ProcessorMetaSupplier metaSupplier) Returns a sink constructed directly from the given Core API processor meta-supplier.static <T> Sink<T>
Sinks.fromProcessor
(String sinkName, ProcessorMetaSupplier metaSupplier, FunctionEx<? super T, ?> partitionKeyFn) Returns a sink constructed directly from the given Core API processor meta-supplier.static <T> Sink<T>
Sinks.jdbc
(String updateQuery, SupplierEx<? extends CommonDataSource> dataSourceSupplier, BiConsumerEx<PreparedStatement, T> bindFn) A shortcut for:static <T> Sink<T>
Sinks.jdbc
(String updateQuery, DataConnectionRef dataConnectionRef, BiConsumerEx<PreparedStatement, T> bindFn) A shortcut for:static <T> Sink<T>
Sinks.jdbc
(String updateQuery, String jdbcUrl, BiConsumerEx<PreparedStatement, T> bindFn) A shortcut for:static <T> Sink<T>
Sinks.jmsQueue
(String queueName, SupplierEx<jakarta.jms.ConnectionFactory> factorySupplier) Convenience forSinks.jmsQueueBuilder(SupplierEx)
.static <T> Sink<T>
Sinks.jmsTopic
(String topicName, SupplierEx<jakarta.jms.ConnectionFactory> factorySupplier) Shortcut for:static <T> Sink<T>
Convenience forSinks.filesBuilder(java.lang.String)
with the UTF-8 charset and with overwriting of existing files.static <T> Sink<T>
Returns a sink that adds the items it receives to the specified HazelcastIList
.static <T> Sink<T>
Returns a sink that adds the items it receives to a HazelcastIList
with the specified name.static <T> Sink<T>
Sinks.logger()
static <T> Sink<T>
Sinks.logger
(FunctionEx<? super T, String> toStringFn) Returns a sink that logs all the data items it receives, at the INFO level to the log categoryWriteLoggerP
.Returns a sink that putsMap.Entry
s it receives into the given HazelcastIMap
.static <T,
K, V> Sink<T> Sinks.map
(IMap<? super K, ? super V> map, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn) Returns a sink that uses the supplied functions to extract the key and value with which to put to given HazelcastIMap
.Returns a sink that putsMap.Entry
s it receives into a HazelcastIMap
with the specified name.static <T,
K, V> Sink<T> Sinks.map
(String mapName, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn) Returns a sink that uses the supplied functions to extract the key and value with which to put to a HazelcastIMap
with the specified name.static <E,
K, V, R> Sink<E> Sinks.mapWithEntryProcessor
(int maxParallelAsyncOps, String mapName, FunctionEx<? super E, ? extends K> toKeyFn, FunctionEx<? super E, ? extends EntryProcessor<K, V, R>> toEntryProcessorFn) Returns a sink that uses the items it receives to createEntryProcessor
s it submits to a HazelcastIMap
with the specified name.static <T,
K, V, R> Sink<T> Sinks.mapWithEntryProcessor
(IMap<? super K, ? super V> map, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends EntryProcessor<K, V, R>> toEntryProcessorFn) Returns a sink that uses the items it receives to createEntryProcessor
s it submits to a HazelcastIMap
with the specified name.static <E,
K, V, R> Sink<E> Sinks.mapWithEntryProcessor
(String mapName, FunctionEx<? super E, ? extends K> toKeyFn, FunctionEx<? super E, ? extends EntryProcessor<K, V, R>> toEntryProcessorFn) Convenience forSinks.mapWithEntryProcessor(int, String, FunctionEx, FunctionEx)
when the maximum number of async operations is not specified.static <T,
K, V> Sink<T> Sinks.mapWithMerging
(IMap<? super K, ? super V> map, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn, BinaryOperatorEx<V> mergeFn) Returns a sink that uses the supplied functions to extract the key and value with which to update a HazelcastIMap
.Sinks.mapWithMerging
(IMap<? super K, V> map, BinaryOperatorEx<V> mergeFn) Convenience forSinks.mapWithMerging(IMap, FunctionEx, FunctionEx, BinaryOperatorEx)
withMap.Entry
as input item.Sinks.mapWithMerging
(String mapName, BinaryOperatorEx<V> mergeFn) Convenience forSinks.mapWithMerging(String, FunctionEx, FunctionEx, BinaryOperatorEx)
withMap.Entry
as input item.static <T,
K, V> Sink<T> Sinks.mapWithMerging
(String mapName, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn, BinaryOperatorEx<V> mergeFn) Returns a sink that uses the supplied functions to extract the key and value with which to update a HazelcastIMap
.Sinks.mapWithUpdating
(IMap<? super K, ? super V> map, BiFunctionEx<? super V, ? super E, ? extends V> updateFn) Convenience forSinks.mapWithUpdating(IMap, FunctionEx, BiFunctionEx)
withMap.Entry
as the input item.static <T,
K, V> Sink<T> Sinks.mapWithUpdating
(IMap<? super K, ? super V> map, FunctionEx<? super T, ? extends K> toKeyFn, BiFunctionEx<? super V, ? super T, ? extends V> updateFn) Returns a sink that uses the supplied key-extracting and value-updating functions to update a HazelcastIMap
.Sinks.mapWithUpdating
(String mapName, BiFunctionEx<? super V, ? super E, ? extends V> updateFn) Convenience forSinks.mapWithUpdating(String, FunctionEx, BiFunctionEx)
withMap.Entry
as the input item.static <T,
K, V> Sink<T> Sinks.mapWithUpdating
(String mapName, FunctionEx<? super T, ? extends K> toKeyFn, BiFunctionEx<? super V, ? super T, ? extends V> updateFn) Returns a sink that uses the supplied key-extracting and value-updating functions to update a HazelcastIMap
.static <T> Sink<T>
Sinks.noop()
Returns a sink which discards all received items.static <T> Sink<T>
Sinks.observable
(Observable<? super T> observable) Returns a sink that publishes to the providedObservable
.static <T> Sink<T>
Sinks.observable
(String name) Returns a sink that publishes to theObservable
with the provided name.static <T> Sink<T>
Sinks.reliableTopic
(ITopic<Object> reliableTopic) Returns a sink which publishes the items it receives to the provided distributed reliable topic.static <T> Sink<T>
Sinks.reliableTopic
(String reliableTopicName) Returns a sink which publishes the items it receives to a distributed reliable topic with the specified name.Sinks.remoteCache
(String cacheName, ClientConfig clientConfig) Returns a sink that putsMap.Entry
s it receives into a HazelcastICache
with the specified name in a remote cluster identified by the suppliedClientConfig
.static <T> Sink<T>
Sinks.remoteList
(String listName, ClientConfig clientConfig) Returns a sink that adds the items it receives to a HazelcastIList
with the specified name in a remote cluster identified by the suppliedClientConfig
.Sinks.remoteMap
(String mapName, ClientConfig clientConfig) Returns a sink that putsMap.Entry
s it receives into a HazelcastIMap
with the specified name in a remote cluster identified by the suppliedClientConfig
.static <T,
K, V> Sink<T> Sinks.remoteMap
(String mapName, ClientConfig clientConfig, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn) Returns a sink that uses the supplied functions to extract the key and value with which to put to a HazelcastIMap
in a remote cluster identified by the suppliedClientConfig
.Sinks.remoteMap
(String mapName, DataConnectionRef dataConnectionRef) The same as theSinks.remoteMap(String, ClientConfig)
method.static <T,
K, V> Sink<T> Sinks.remoteMap
(String mapName, DataConnectionRef dataConnectionRef, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn) The same as theSinks.remoteMap(String, ClientConfig, FunctionEx, FunctionEx)
method.static <E,
K, V, R> Sink<E> Sinks.remoteMapWithEntryProcessor
(String mapName, ClientConfig clientConfig, FunctionEx<? super E, ? extends K> toKeyFn, FunctionEx<? super E, ? extends EntryProcessor<K, V, R>> toEntryProcessorFn) Returns a sink equivalent toSinks.mapWithEntryProcessor(java.lang.String, com.hazelcast.function.FunctionEx<? super E, ? extends K>, com.hazelcast.function.FunctionEx<? super E, ? extends com.hazelcast.map.EntryProcessor<K, V, R>>)
, but for a map in a remote Hazelcast cluster identified by the suppliedClientConfig
.static <E,
K, V, R> Sink<E> Sinks.remoteMapWithEntryProcessor
(String mapName, DataConnectionRef dataConnectionRef, FunctionEx<? super E, ? extends K> toKeyFn, FunctionEx<? super E, ? extends EntryProcessor<K, V, R>> toEntryProcessorFn) The same as theSinks.remoteMapWithEntryProcessor(String, ClientConfig, FunctionEx, FunctionEx)
method.Sinks.remoteMapWithMerging
(String mapName, ClientConfig clientConfig, BinaryOperatorEx<V> mergeFn) Convenience forSinks.remoteMapWithMerging(java.lang.String, com.hazelcast.client.config.ClientConfig, com.hazelcast.function.BinaryOperatorEx<V>)
withMap.Entry
as input item.static <T,
K, V> Sink<T> Sinks.remoteMapWithMerging
(String mapName, ClientConfig clientConfig, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn, BinaryOperatorEx<V> mergeFn) Returns a sink equivalent toSinks.mapWithMerging(String, BinaryOperatorEx)
, but for a map in a remote Hazelcast cluster identified by the suppliedClientConfig
.Sinks.remoteMapWithMerging
(String mapName, DataConnectionRef dataConnectionRef, BinaryOperatorEx<V> mergeFn) The same as theSinks.remoteMapWithMerging(String, ClientConfig, BinaryOperatorEx)
method.static <T,
K, V> Sink<T> Sinks.remoteMapWithMerging
(String mapName, DataConnectionRef dataConnectionRef, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn, BinaryOperatorEx<V> mergeFn) The same as theSinks.remoteMapWithMerging(String, ClientConfig, FunctionEx, FunctionEx, BinaryOperatorEx)
method.Sinks.remoteMapWithUpdating
(String mapName, ClientConfig clientConfig, BiFunctionEx<? super V, ? super E, ? extends V> updateFn) static <T,
K, V> Sink<T> Sinks.remoteMapWithUpdating
(String mapName, ClientConfig clientConfig, FunctionEx<? super T, ? extends K> toKeyFn, BiFunctionEx<? super V, ? super T, ? extends V> updateFn) Returns a sink equivalent toSinks.mapWithUpdating(java.lang.String, com.hazelcast.function.FunctionEx<? super T, ? extends K>, com.hazelcast.function.BiFunctionEx<? super V, ? super T, ? extends V>)
, but for a map in a remote Hazelcast cluster identified by the suppliedClientConfig
.Sinks.remoteMapWithUpdating
(String mapName, DataConnectionRef dataConnectionRef, BiFunctionEx<? super V, ? super E, ? extends V> updateFn) The same as theSinks.remoteMapWithUpdating(String, ClientConfig, BiFunctionEx)
method.static <T,
K, V> Sink<T> Sinks.remoteMapWithUpdating
(String mapName, DataConnectionRef dataConnectionRef, FunctionEx<? super T, ? extends K> toKeyFn, BiFunctionEx<? super V, ? super T, ? extends V> updateFn) The same as theSinks.remoteMapWithUpdating(String, ClientConfig, BiFunctionEx)
method.static <T> Sink<T>
Sinks.remoteReliableTopic
(String reliableTopicName, ClientConfig clientConfig) Returns a sink which publishes the items it receives to a distributed reliable topic with the provided name in a remote cluster identified by the suppliedClientConfig
.static <T> Sink<T>
Convenience forSinks.socket(String, int, FunctionEx, Charset)
withObject.toString
as the conversion function and UTF-8 as the charset.static <T> Sink<T>
Sinks.socket
(String host, int port, FunctionEx<? super T, ? extends String> toStringFn) Convenience forSinks.socket(String, int, FunctionEx, Charset)
with UTF-8 as the charset.static <T> Sink<T>
Sinks.socket
(String host, int port, FunctionEx<? super T, ? extends String> toStringFn, Charset charset) Returns a sink that connects to the specified TCP socket and writes to it a string representation of the items it receives.Modifier and TypeMethodDescriptionAttaches a sink stage, one that accepts data but doesn't emit any.<T> SinkStage
Pipeline.writeTo
(Sink<? super T> sink, GeneralStage<? extends T> stage0, GeneralStage<? extends T> stage1, GeneralStage<? extends T>... moreStages) Attaches the supplied sink to two or more pipeline stages. -
Uses of Sink in com.hazelcast.jet.pipeline.test
Modifier and TypeMethodDescriptionstatic <T> Sink<T>
AssertionSinks.assertAnyOrder
(String message, Collection<? extends T> expected) Asserts that the previous stage emitted the expected items in any order, but nothing else.static <T> Sink<T>
AssertionSinks.assertAnyOrder
(Collection<? extends T> expected) Asserts that the previous stage emitted the expected items in any order, but nothing else.static <T> Sink<T>
AssertionSinks.assertCollected
(ConsumerEx<? super List<T>> assertFn) Collects all the received items in a list and once the upstream stage is completed it executes the assertion supplied byassertFn
.static <T> Sink<T>
AssertionSinks.assertCollectedEventually
(int timeoutSeconds, ConsumerEx<? super List<T>> assertFn) Collects all the received items into a list and runs theassertFn
every time a new item is received.static <T> Sink<T>
AssertionSinks.assertContains
(String message, Collection<? extends T> expected) Asserts that the previous stage emitted all the given items in any order.static <T> Sink<T>
AssertionSinks.assertOrdered
(String message, Collection<? extends T> expected) Asserts that the previous stage emitted the exact sequence of expected items and nothing else.static <T> Sink<T>
AssertionSinks.assertOrdered
(Collection<? extends T> expected) Asserts that the previous stage emitted the exact sequence of expected items and nothing else.AssertionSinkBuilder.build()
Creates and returns theSink
with the components you supplied to this builder. -
Uses of Sink in com.hazelcast.jet.s3
Modifier and TypeMethodDescriptionstatic <T> Sink<? super T>
S3Sinks.s3
(String bucketName, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier) Convenience forS3Sinks.s3(String, String, Charset, SupplierEx, FunctionEx)
UsesObject.toString()
to convert the items to lines.static <T> Sink<? super T>
S3Sinks.s3
(String bucketName, String prefix, Charset charset, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier, FunctionEx<? super T, String> toStringFn) Creates an AWS S3Sink
which writes items to files into the given bucket.