Uses of Interface
com.hazelcast.jet.pipeline.Sink
Packages that use Sink
Package
Description
Apache Avro file read/write support for Hazelcast Jet.
Contains source/sink connectors that deal with Change Data Capture (CDC)
events from various databases as well as a generic connector for Debezium
CDC sources.
Contains sources and sinks for Elasticsearch 7
Apache Hadoop read/write support for Hazelcast Jet.
Apache Kafka reader/writer support for Hazelcast Jet.
Amazon Kinesis Data Streams producer/consumer support for Hazelcast Jet.
Contains sources and sinks for MongoDB.
The Pipeline API is Jet's high-level API to build and execute
distributed computation jobs.
This package contains various mock sources to help with pipeline testing
and development.
AWS S3 read/write support for Hazelcast Jet.
-
Uses of Sink in com.hazelcast.jet.avro
Methods in com.hazelcast.jet.avro that return SinkModifier and TypeMethodDescriptionstatic <R> Sink<R>Convenience forAvroSinks.files(String, Schema, SupplierEx)which uses eitherSpecificDatumWriterorReflectDatumWriterdepending on the suppliedrecordClass.static Sink<org.apache.avro.generic.IndexedRecord>Convenience forAvroSinks.files(String, Schema, SupplierEx)which usesGenericDatumWriter.static <R> Sink<R>AvroSinks.files(String directoryName, org.apache.avro.Schema schema, SupplierEx<org.apache.avro.io.DatumWriter<R>> datumWriterSupplier) Returns a sink that that writes the items it receives to Apache Avro files. -
Uses of Sink in com.hazelcast.jet.cdc
Methods in com.hazelcast.jet.cdc that return SinkModifier and TypeMethodDescriptionstatic <K,V> Sink<ChangeRecord> CdcSinks.map(IMap<? super K, ? super V> map, FunctionEx<? super ChangeRecord, ? extends K> keyFn, FunctionEx<? super ChangeRecord, ? extends V> valueFn) Returns a sink that applies the changes described by a Change Data Capture (CDC) stream to anIMap.static <K,V> Sink<ChangeRecord> CdcSinks.map(String mapName, FunctionEx<? super ChangeRecord, ? extends K> keyFn, FunctionEx<? super ChangeRecord, ? extends V> valueFn) Returns a sink that applies the changes described by a Change Data Capture (CDC) stream to anIMap.static <K,V> Sink<ChangeRecord> CdcSinks.remoteMap(String mapName, ClientConfig clientConfig, FunctionEx<? super ChangeRecord, ? extends K> keyFn, FunctionEx<? super ChangeRecord, ? extends V> valueFn) Returns a sink equivalent toCdcSinks.map(java.lang.String, com.hazelcast.function.FunctionEx<? super com.hazelcast.jet.cdc.ChangeRecord, ? extends K>, com.hazelcast.function.FunctionEx<? super com.hazelcast.jet.cdc.ChangeRecord, ? extends V>), but for a map in a remote Hazelcast cluster identified by the suppliedClientConfig. -
Uses of Sink in com.hazelcast.jet.elastic
Methods in com.hazelcast.jet.elastic that return SinkModifier and TypeMethodDescriptionElasticSinkBuilder.build()Create a sink that writes data into Elasticsearch based on this builder configurationstatic <T> Sink<T>ElasticSinks.elastic(FunctionEx<? super T, ? extends org.elasticsearch.action.DocWriteRequest<?>> mapToRequestFn) Creates an Elasticsearch sink, uses a local instance of Elasticsearchstatic <T> Sink<T>ElasticSinks.elastic(SupplierEx<org.elasticsearch.client.RestClientBuilder> clientFn, FunctionEx<? super T, ? extends org.elasticsearch.action.DocWriteRequest<?>> mapToRequestFn) Creates an Elasticsearch sink, uses a client obtained from clientFn and maps items using given mapToRequestFn -
Uses of Sink in com.hazelcast.jet.hadoop
Methods in com.hazelcast.jet.hadoop that return SinkModifier and TypeMethodDescriptionHadoopSinks.outputFormat(org.apache.hadoop.conf.Configuration configuration) Convenience forHadoopSinks.outputFormat(Configuration, FunctionEx, FunctionEx)which expectsMap.Entry<K, V>as input and extracts its key and value parts to be written to HDFS.static <E,K, V> Sink<E> HadoopSinks.outputFormat(org.apache.hadoop.conf.Configuration configuration, FunctionEx<? super E, K> extractKeyF, FunctionEx<? super E, V> extractValueF) Returns a sink that writes to Apache Hadoop HDFS. -
Uses of Sink in com.hazelcast.jet.kafka
Methods in com.hazelcast.jet.kafka that return SinkModifier and TypeMethodDescriptionKafkaSinks.Builder.build()Builds the Sink object that you pass to theGeneralStage.writeTo(Sink)method.static <E,K, V> Sink<E> KafkaSinks.kafka(DataConnectionRef dataConnectionRef, FunctionEx<? super E, org.apache.kafka.clients.producer.ProducerRecord<K, V>> toRecordFn) Returns a sink that publishes messages to Apache Kafka topic(s).KafkaSinks.kafka(DataConnectionRef dataConnectionRef, String topic) Convenience forKafkaSinks.kafka(DataConnectionRef, String, FunctionEx, FunctionEx)which expectsMap.Entry<K, V>as input and extracts its key and value parts to be published to Kafka.static <E,K, V> Sink<E> KafkaSinks.kafka(DataConnectionRef dataConnectionRef, String topic, FunctionEx<? super E, K> extractKeyFn, FunctionEx<? super E, V> extractValueFn) Convenience forKafkaSinks.kafka(Properties, FunctionEx)which creates aProducerRecordusing the given topic and the given key and value mapping functionsstatic <E,K, V> Sink<E> KafkaSinks.kafka(DataConnectionRef dataConnectionRef, Properties properties, String topic, FunctionEx<? super E, K> extractKeyFn, FunctionEx<? super E, V> extractValueFn) Convenience forKafkaSinks.kafka(Properties, FunctionEx)which creates aProducerRecordusing the given topic and the given key and value mapping functions with additional properties availablestatic <E,K, V> Sink<E> KafkaSinks.kafka(Properties properties, FunctionEx<? super E, org.apache.kafka.clients.producer.ProducerRecord<K, V>> toRecordFn) Returns a sink that publishes messages to Apache Kafka topic(s).KafkaSinks.kafka(Properties properties, String topic) Convenience forKafkaSinks.kafka(Properties, String, FunctionEx, FunctionEx)which expectsMap.Entry<K, V>as input and extracts its key and value parts to be published to Kafka.static <E,K, V> Sink<E> KafkaSinks.kafka(Properties properties, String topic, FunctionEx<? super E, K> extractKeyFn, FunctionEx<? super E, V> extractValueFn) Convenience forKafkaSinks.kafka(Properties, FunctionEx)which creates aProducerRecordusing the given topic and the given key and value mapping functions -
Uses of Sink in com.hazelcast.jet.kinesis
Methods in com.hazelcast.jet.kinesis that return Sink -
Uses of Sink in com.hazelcast.jet.mongodb
Methods in com.hazelcast.jet.mongodb that return SinkModifier and TypeMethodDescriptionMongoSinkBuilder.build()Creates and returns the MongoDBSinkwith the components you supplied to this builder.static Sink<org.bson.Document>MongoSinks.mongodb(DataConnectionRef dataConnectionRef, String database, String collection) static Sink<org.bson.Document> -
Uses of Sink in com.hazelcast.jet.pipeline
Methods in com.hazelcast.jet.pipeline that return SinkModifier and TypeMethodDescriptionFileSinkBuilder.build()Creates and returns the fileSinkwith the supplied components.JdbcSinkBuilder.build()Creates and returns the JDBCSinkwith the supplied components.JmsSinkBuilder.build()Creates and returns the JMSSinkwith the supplied components.MapSinkBuilder.build()Build the sink.MapSinkEntryProcessorBuilder.build()SinkBuilder.build()Creates and returns theSinkwith the components you supplied to this builder.Returns a sink that putsMap.Entrys it receives into a HazelcastICachewith the specified name.static <T> Sink<T>Convenience forSinks.filesBuilder(java.lang.String)with the UTF-8 charset and with overwriting of existing files.static <T> Sink<T>Sinks.fromProcessor(String sinkName, ProcessorMetaSupplier metaSupplier) Returns a sink constructed directly from the given Core API processor meta-supplier.static <T> Sink<T>Sinks.fromProcessor(String sinkName, ProcessorMetaSupplier metaSupplier, FunctionEx<? super T, ?> partitionKeyFn) Returns a sink constructed directly from the given Core API processor meta-supplier.static <T> Sink<T>Sinks.jdbc(String updateQuery, SupplierEx<? extends CommonDataSource> dataSourceSupplier, BiConsumerEx<PreparedStatement, T> bindFn) A shortcut for:static <T> Sink<T>Sinks.jdbc(String updateQuery, DataConnectionRef dataConnectionRef, BiConsumerEx<PreparedStatement, T> bindFn) A shortcut for:static <T> Sink<T>Sinks.jdbc(String updateQuery, String jdbcUrl, BiConsumerEx<PreparedStatement, T> bindFn) A shortcut for:static <T> Sink<T>Sinks.jmsQueue(String queueName, SupplierEx<jakarta.jms.ConnectionFactory> factorySupplier) Convenience forSinks.jmsQueueBuilder(SupplierEx).static <T> Sink<T>Sinks.jmsTopic(String topicName, SupplierEx<jakarta.jms.ConnectionFactory> factorySupplier) Shortcut for:static <T> Sink<T>Convenience forSinks.filesBuilder(java.lang.String)with the UTF-8 charset and with overwriting of existing files.static <T> Sink<T>Returns a sink that adds the items it receives to the specified HazelcastIList.static <T> Sink<T>Returns a sink that adds the items it receives to a HazelcastIListwith the specified name.static <T> Sink<T>Sinks.logger()static <T> Sink<T>Sinks.logger(FunctionEx<? super T, String> toStringFn) Returns a sink that logs all the data items it receives, at the INFO level to the log categoryWriteLoggerP.Returns a sink that putsMap.Entrys it receives into the given HazelcastIMap.static <T,K, V> Sink<T> Sinks.map(IMap<? super K, ? super V> map, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn) Returns a sink that uses the supplied functions to extract the key and value with which to put to given HazelcastIMap.Returns a sink that putsMap.Entrys it receives into a HazelcastIMapwith the specified name.static <T,K, V> Sink<T> Sinks.map(String mapName, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn) Returns a sink that uses the supplied functions to extract the key and value with which to put to a HazelcastIMapwith the specified name.static <E,K, V, R> Sink<E> Sinks.mapWithEntryProcessor(int maxParallelAsyncOps, String mapName, FunctionEx<? super E, ? extends K> toKeyFn, FunctionEx<? super E, ? extends EntryProcessor<K, V, R>> toEntryProcessorFn) Returns a sink that uses the items it receives to createEntryProcessors it submits to a HazelcastIMapwith the specified name.static <T,K, V, R> Sink<T> Sinks.mapWithEntryProcessor(IMap<? super K, ? super V> map, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends EntryProcessor<K, V, R>> toEntryProcessorFn) Returns a sink that uses the items it receives to createEntryProcessors it submits to a HazelcastIMapwith the specified name.static <E,K, V, R> Sink<E> Sinks.mapWithEntryProcessor(String mapName, FunctionEx<? super E, ? extends K> toKeyFn, FunctionEx<? super E, ? extends EntryProcessor<K, V, R>> toEntryProcessorFn) Convenience forSinks.mapWithEntryProcessor(int, String, FunctionEx, FunctionEx)when the maximum number of async operations is not specified.static <T,K, V> Sink<T> Sinks.mapWithMerging(IMap<? super K, ? super V> map, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn, BinaryOperatorEx<V> mergeFn) Returns a sink that uses the supplied functions to extract the key and value with which to update a HazelcastIMap.Sinks.mapWithMerging(IMap<? super K, V> map, BinaryOperatorEx<V> mergeFn) Convenience forSinks.mapWithMerging(IMap, FunctionEx, FunctionEx, BinaryOperatorEx)withMap.Entryas input item.Sinks.mapWithMerging(String mapName, BinaryOperatorEx<V> mergeFn) Convenience forSinks.mapWithMerging(String, FunctionEx, FunctionEx, BinaryOperatorEx)withMap.Entryas input item.static <T,K, V> Sink<T> Sinks.mapWithMerging(String mapName, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn, BinaryOperatorEx<V> mergeFn) Returns a sink that uses the supplied functions to extract the key and value with which to update a HazelcastIMap.Sinks.mapWithUpdating(IMap<? super K, ? super V> map, BiFunctionEx<? super V, ? super E, ? extends V> updateFn) Convenience forSinks.mapWithUpdating(IMap, FunctionEx, BiFunctionEx)withMap.Entryas the input item.static <T,K, V> Sink<T> Sinks.mapWithUpdating(IMap<? super K, ? super V> map, FunctionEx<? super T, ? extends K> toKeyFn, BiFunctionEx<? super V, ? super T, ? extends V> updateFn) Returns a sink that uses the supplied key-extracting and value-updating functions to update a HazelcastIMap.Sinks.mapWithUpdating(String mapName, BiFunctionEx<? super V, ? super E, ? extends V> updateFn) Convenience forSinks.mapWithUpdating(String, FunctionEx, BiFunctionEx)withMap.Entryas the input item.static <T,K, V> Sink<T> Sinks.mapWithUpdating(String mapName, FunctionEx<? super T, ? extends K> toKeyFn, BiFunctionEx<? super V, ? super T, ? extends V> updateFn) Returns a sink that uses the supplied key-extracting and value-updating functions to update a HazelcastIMap.static <T> Sink<T>Sinks.noop()Returns a sink which discards all received items.static <T> Sink<T>Sinks.observable(Observable<? super T> observable) Returns a sink that publishes to the providedObservable.static <T> Sink<T>Sinks.observable(String name) Returns a sink that publishes to theObservablewith the provided name.static <T> Sink<T>Sinks.reliableTopic(ITopic<Object> reliableTopic) Returns a sink which publishes the items it receives to the provided distributed reliable topic.static <T> Sink<T>Sinks.reliableTopic(String reliableTopicName) Returns a sink which publishes the items it receives to a distributed reliable topic with the specified name.Sinks.remoteCache(String cacheName, ClientConfig clientConfig) Returns a sink that putsMap.Entrys it receives into a HazelcastICachewith the specified name in a remote cluster identified by the suppliedClientConfig.static <T> Sink<T>Sinks.remoteList(String listName, ClientConfig clientConfig) Returns a sink that adds the items it receives to a HazelcastIListwith the specified name in a remote cluster identified by the suppliedClientConfig.Sinks.remoteMap(String mapName, ClientConfig clientConfig) Returns a sink that putsMap.Entrys it receives into a HazelcastIMapwith the specified name in a remote cluster identified by the suppliedClientConfig.static <T,K, V> Sink<T> Sinks.remoteMap(String mapName, ClientConfig clientConfig, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn) Returns a sink that uses the supplied functions to extract the key and value with which to put to a HazelcastIMapin a remote cluster identified by the suppliedClientConfig.Sinks.remoteMap(String mapName, DataConnectionRef dataConnectionRef) The same as theSinks.remoteMap(String, ClientConfig)method.static <T,K, V> Sink<T> Sinks.remoteMap(String mapName, DataConnectionRef dataConnectionRef, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn) The same as theSinks.remoteMap(String, ClientConfig, FunctionEx, FunctionEx)method.static <E,K, V, R> Sink<E> Sinks.remoteMapWithEntryProcessor(String mapName, ClientConfig clientConfig, FunctionEx<? super E, ? extends K> toKeyFn, FunctionEx<? super E, ? extends EntryProcessor<K, V, R>> toEntryProcessorFn) Returns a sink equivalent toSinks.mapWithEntryProcessor(java.lang.String, com.hazelcast.function.FunctionEx<? super E, ? extends K>, com.hazelcast.function.FunctionEx<? super E, ? extends com.hazelcast.map.EntryProcessor<K, V, R>>), but for a map in a remote Hazelcast cluster identified by the suppliedClientConfig.static <E,K, V, R> Sink<E> Sinks.remoteMapWithEntryProcessor(String mapName, DataConnectionRef dataConnectionRef, FunctionEx<? super E, ? extends K> toKeyFn, FunctionEx<? super E, ? extends EntryProcessor<K, V, R>> toEntryProcessorFn) The same as theSinks.remoteMapWithEntryProcessor(String, ClientConfig, FunctionEx, FunctionEx)method.Sinks.remoteMapWithMerging(String mapName, ClientConfig clientConfig, BinaryOperatorEx<V> mergeFn) Convenience forSinks.remoteMapWithMerging(java.lang.String, com.hazelcast.client.config.ClientConfig, com.hazelcast.function.BinaryOperatorEx<V>)withMap.Entryas input item.static <T,K, V> Sink<T> Sinks.remoteMapWithMerging(String mapName, ClientConfig clientConfig, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn, BinaryOperatorEx<V> mergeFn) Returns a sink equivalent toSinks.mapWithMerging(String, BinaryOperatorEx), but for a map in a remote Hazelcast cluster identified by the suppliedClientConfig.Sinks.remoteMapWithMerging(String mapName, DataConnectionRef dataConnectionRef, BinaryOperatorEx<V> mergeFn) The same as theSinks.remoteMapWithMerging(String, ClientConfig, BinaryOperatorEx)method.static <T,K, V> Sink<T> Sinks.remoteMapWithMerging(String mapName, DataConnectionRef dataConnectionRef, FunctionEx<? super T, ? extends K> toKeyFn, FunctionEx<? super T, ? extends V> toValueFn, BinaryOperatorEx<V> mergeFn) The same as theSinks.remoteMapWithMerging(String, ClientConfig, FunctionEx, FunctionEx, BinaryOperatorEx)method.Sinks.remoteMapWithUpdating(String mapName, ClientConfig clientConfig, BiFunctionEx<? super V, ? super E, ? extends V> updateFn) static <T,K, V> Sink<T> Sinks.remoteMapWithUpdating(String mapName, ClientConfig clientConfig, FunctionEx<? super T, ? extends K> toKeyFn, BiFunctionEx<? super V, ? super T, ? extends V> updateFn) Returns a sink equivalent toSinks.mapWithUpdating(java.lang.String, com.hazelcast.function.FunctionEx<? super T, ? extends K>, com.hazelcast.function.BiFunctionEx<? super V, ? super T, ? extends V>), but for a map in a remote Hazelcast cluster identified by the suppliedClientConfig.Sinks.remoteMapWithUpdating(String mapName, DataConnectionRef dataConnectionRef, BiFunctionEx<? super V, ? super E, ? extends V> updateFn) The same as theSinks.remoteMapWithUpdating(String, ClientConfig, BiFunctionEx)method.static <T,K, V> Sink<T> Sinks.remoteMapWithUpdating(String mapName, DataConnectionRef dataConnectionRef, FunctionEx<? super T, ? extends K> toKeyFn, BiFunctionEx<? super V, ? super T, ? extends V> updateFn) The same as theSinks.remoteMapWithUpdating(String, ClientConfig, BiFunctionEx)method.static <T> Sink<T>Sinks.remoteReliableTopic(String reliableTopicName, ClientConfig clientConfig) Returns a sink which publishes the items it receives to a distributed reliable topic with the provided name in a remote cluster identified by the suppliedClientConfig.static <T> Sink<T>Convenience forSinks.socket(String, int, FunctionEx, Charset)withObject.toStringas the conversion function and UTF-8 as the charset.static <T> Sink<T>Sinks.socket(String host, int port, FunctionEx<? super T, ? extends String> toStringFn) Convenience forSinks.socket(String, int, FunctionEx, Charset)with UTF-8 as the charset.static <T> Sink<T>Sinks.socket(String host, int port, FunctionEx<? super T, ? extends String> toStringFn, Charset charset) Returns a sink that connects to the specified TCP socket and writes to it a string representation of the items it receives.Methods in com.hazelcast.jet.pipeline with parameters of type SinkModifier and TypeMethodDescriptionAttaches a sink stage, one that accepts data but doesn't emit any.<T> SinkStagePipeline.writeTo(Sink<? super T> sink, GeneralStage<? extends T> stage0, GeneralStage<? extends T> stage1, GeneralStage<? extends T>... moreStages) Attaches the supplied sink to two or more pipeline stages. -
Uses of Sink in com.hazelcast.jet.pipeline.test
Methods in com.hazelcast.jet.pipeline.test that return SinkModifier and TypeMethodDescriptionstatic <T> Sink<T>AssertionSinks.assertAnyOrder(String message, Collection<? extends T> expected) Asserts that the previous stage emitted the expected items in any order, but nothing else.static <T> Sink<T>AssertionSinks.assertAnyOrder(Collection<? extends T> expected) Asserts that the previous stage emitted the expected items in any order, but nothing else.static <T> Sink<T>AssertionSinks.assertCollected(ConsumerEx<? super List<T>> assertFn) Collects all the received items in a list and once the upstream stage is completed it executes the assertion supplied byassertFn.static <T> Sink<T>AssertionSinks.assertCollectedEventually(int timeoutSeconds, ConsumerEx<? super List<T>> assertFn) Collects all the received items into a list and runs theassertFnevery time a new item is received.static <T> Sink<T>AssertionSinks.assertContains(String message, Collection<? extends T> expected) Asserts that the previous stage emitted all the given items in any order.static <T> Sink<T>AssertionSinks.assertOrdered(String message, Collection<? extends T> expected) Asserts that the previous stage emitted the exact sequence of expected items and nothing else.static <T> Sink<T>AssertionSinks.assertOrdered(Collection<? extends T> expected) Asserts that the previous stage emitted the exact sequence of expected items and nothing else.AssertionSinkBuilder.build()Creates and returns theSinkwith the components you supplied to this builder. -
Uses of Sink in com.hazelcast.jet.s3
Methods in com.hazelcast.jet.s3 that return SinkModifier and TypeMethodDescriptionstatic <T> Sink<? super T>S3Sinks.s3(String bucketName, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier) Convenience forS3Sinks.s3(String, String, Charset, SupplierEx, FunctionEx)UsesObject.toString()to convert the items to lines.static <T> Sink<? super T>S3Sinks.s3(String bucketName, String prefix, Charset charset, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier, FunctionEx<? super T, String> toStringFn) Creates an AWS S3Sinkwhich writes items to files into the given bucket.