Uses of Interface
com.hazelcast.jet.pipeline.StreamSource
Packages that use StreamSource
Package
Description
Contains source/sink connectors that deal with Change Data Capture (CDC)
events from various databases as well as a generic connector for Debezium
CDC sources.
Contains connectors for change data capture events from MySQL
databases.
Contains connectors for change data capture events from PostgreSQL
databases.
Apache Kafka reader/writer support for Hazelcast Jet.
Contains a generic Kafka Connect source provides ability to plug any Kafka
Connect source for data ingestion to Jet pipelines.
Amazon Kinesis Data Streams producer/consumer support for Hazelcast Jet.
Contains sources and sinks for MongoDB.
The Pipeline API is Jet's high-level API to build and execute
distributed computation jobs.
This package contains various mock sources to help with pipeline testing
and development.
-
Uses of StreamSource in com.hazelcast.jet.cdc
Methods in com.hazelcast.jet.cdc that return StreamSourceModifier and TypeMethodDescriptionDebeziumCdcSources.Builder.build()
Returns the CDC source based on the properties set. -
Uses of StreamSource in com.hazelcast.jet.cdc.mysql
Methods in com.hazelcast.jet.cdc.mysql that return StreamSourceModifier and TypeMethodDescriptionMySqlCdcSources.Builder.build()
Returns the source based on the properties set so far. -
Uses of StreamSource in com.hazelcast.jet.cdc.postgres
Methods in com.hazelcast.jet.cdc.postgres that return StreamSourceModifier and TypeMethodDescriptionPostgresCdcSources.Builder.build()
Returns the source based on the properties set so far. -
Uses of StreamSource in com.hazelcast.jet.kafka
Methods in com.hazelcast.jet.kafka that return StreamSourceModifier and TypeMethodDescriptionstatic <K,
V, T> StreamSource<T> KafkaSources.kafka
(DataConnectionRef dataConnectionRef, FunctionEx<org.apache.kafka.clients.consumer.ConsumerRecord<K, V>, T> projectionFn, String... topics) Returns a source that consumes one or more Apache Kafka topics and emits items from them asMap.Entry
instances.static <K,
V> StreamSource<Map.Entry<K, V>> KafkaSources.kafka
(DataConnectionRef dataConnectionRef, String... topics) Convenience forKafkaSources.kafka(DataConnectionRef, FunctionEx, String...)
wrapping the output inMap.Entry
.static <K,
V, T> StreamSource<T> KafkaSources.kafka
(Properties properties, FunctionEx<org.apache.kafka.clients.consumer.ConsumerRecord<K, V>, T> projectionFn, TopicsConfig topicsConfig) Returns a source that consumes one or more Apache Kafka topics and emits items from them asMap.Entry
instances.static <K,
V, T> StreamSource<T> KafkaSources.kafka
(Properties properties, FunctionEx<org.apache.kafka.clients.consumer.ConsumerRecord<K, V>, T> projectionFn, String... topics) Returns a source that consumes one or more Apache Kafka topics and emits items from them asMap.Entry
instances.static <K,
V> StreamSource<Map.Entry<K, V>> KafkaSources.kafka
(Properties properties, String... topics) Convenience forKafkaSources.kafka(Properties, FunctionEx, String...)
wrapping the output inMap.Entry
. -
Uses of StreamSource in com.hazelcast.jet.kafka.connect
Methods in com.hazelcast.jet.kafka.connect that return StreamSourceModifier and TypeMethodDescriptionstatic StreamSource<org.apache.kafka.connect.source.SourceRecord>
KafkaConnectSources.connect
(Properties properties) A generic Kafka Connect source provides ability to plug any Kafka Connect source for data ingestion to Jet pipelines.static <T> StreamSource<T>
KafkaConnectSources.connect
(Properties properties, FunctionEx<org.apache.kafka.connect.source.SourceRecord, T> projectionFn) A generic Kafka Connect source provides ability to plug any Kafka Connect source for data ingestion to Jet pipelines. -
Uses of StreamSource in com.hazelcast.jet.kinesis
Methods in com.hazelcast.jet.kinesis that return StreamSourceModifier and TypeMethodDescriptionKinesisSources.Builder.build()
Constructs the source based on the options provided so far. -
Uses of StreamSource in com.hazelcast.jet.mongodb
Methods in com.hazelcast.jet.mongodb that return StreamSourceModifier and TypeMethodDescriptionMongoSourceBuilder.Stream.build()
Creates and returns the MongoDBStreamSource
which watches the given collection.static StreamSource<? extends org.bson.Document>
MongoSources.stream
(String connectionString, String database, String collection, org.bson.Document filter, org.bson.Document projection) Returns a MongoDB stream source which watches the changes on the collection. -
Uses of StreamSource in com.hazelcast.jet.pipeline
Methods in com.hazelcast.jet.pipeline that return StreamSourceModifier and TypeMethodDescriptionStreamSource<jakarta.jms.Message>
JmsSourceBuilder.build()
Convenience forJmsSourceBuilder.build(FunctionEx)
.<T> StreamSource<T>
JmsSourceBuilder.build
(FunctionEx<? super jakarta.jms.Message, ? extends T> projectionFn) Creates and returns the JMSStreamSource
with the supplied components and the projection functionprojectionFn
.SourceBuilder.Stream.build()
Builds and returns the unbounded stream source.SourceBuilder.TimestampedStream.build()
Builds and returns the timestamped stream source.FileSourceBuilder.buildWatcher()
Convenience forFileSourceBuilder.buildWatcher(BiFunctionEx)
.<T> StreamSource<T>
FileSourceBuilder.buildWatcher
(BiFunctionEx<String, String, ? extends T> mapOutputFn) Builds a source that emits a stream of lines of text coming from files in the watched directory (but not its subdirectories).static <K,
V> StreamSource<Map.Entry<K, V>> Sources.cacheJournal
(String cacheName, JournalInitialPosition initialPos) Convenience forSources.cacheJournal(String, JournalInitialPosition, FunctionEx, PredicateEx)
which will pass onlyCREATED
andUPDATED
events and will project the event's key and new value into aMap.Entry
.static <T,
K, V> StreamSource<T> Sources.cacheJournal
(String cacheName, JournalInitialPosition initialPos, FunctionEx<? super EventJournalCacheEvent<K, V>, ? extends T> projectionFn, PredicateEx<? super EventJournalCacheEvent<K, V>> predicateFn) Returns a source that will stream theEventJournalCacheEvent
events of a HazelcastICache
with the specified name.static StreamSource<String>
Sources.fileWatcher
(String watchedDirectory) A source to stream lines added to files in a directory.static StreamSource<jakarta.jms.Message>
Sources.jmsQueue
(SupplierEx<? extends jakarta.jms.ConnectionFactory> factorySupplier, String name) Deprecated.see Sources.jmsQueue(String, SupplierEx).static StreamSource<jakarta.jms.Message>
Sources.jmsQueue
(String name, SupplierEx<? extends jakarta.jms.ConnectionFactory> factorySupplier) Shortcut equivalent to:static StreamSource<jakarta.jms.Message>
Sources.jmsTopic
(SupplierEx<? extends jakarta.jms.ConnectionFactory> factorySupplier, String name) Deprecated.static StreamSource<jakarta.jms.Message>
Sources.jmsTopic
(String name, SupplierEx<? extends jakarta.jms.ConnectionFactory> factorySupplier) Shortcut equivalent to:static StreamSource<Map<String,
Object>> Sources.jsonWatcher
(String watchedDirectory) Convenience forSources.jsonWatcher(String, Class)
which converts each line appended to theMap
representation of the JSON string.static <T> StreamSource<T>
Sources.jsonWatcher
(String watchedDirectory, Class<T> type) A source to stream lines added to files in a directory.static <K,
V> StreamSource<Map.Entry<K, V>> Sources.mapJournal
(IMap<? extends K, ? extends V> map, JournalInitialPosition initialPos) Convenience forSources.mapJournal(IMap, JournalInitialPosition, FunctionEx, PredicateEx)
which will pass onlyADDED
andUPDATED
events and will project the event's key and new value into aMap.Entry
.static <T,
K, V> StreamSource<T> Sources.mapJournal
(IMap<? extends K, ? extends V> map, JournalInitialPosition initialPos, FunctionEx<? super EventJournalMapEvent<K, V>, ? extends T> projectionFn, PredicateEx<? super EventJournalMapEvent<K, V>> predicateFn) Returns a source that will streamEventJournalMapEvent
s of the given HazelcastIMap
.static <K,
V> StreamSource<Map.Entry<K, V>> Sources.mapJournal
(String mapName, JournalInitialPosition initialPos) Convenience forSources.mapJournal(String, JournalInitialPosition, FunctionEx, PredicateEx)
which will pass onlyADDED
andUPDATED
events and will project the event's key and new value into aMap.Entry
.static <T,
K, V> StreamSource<T> Sources.mapJournal
(String mapName, JournalInitialPosition initialPos, FunctionEx<? super EventJournalMapEvent<K, V>, ? extends T> projectionFn, PredicateEx<? super EventJournalMapEvent<K, V>> predicateFn) Returns a source that will streamEventJournalMapEvent
s of the HazelcastIMap
with the specified name.static <K,
V> StreamSource<Map.Entry<K, V>> Sources.remoteCacheJournal
(String cacheName, ClientConfig clientConfig, JournalInitialPosition initialPos) Convenience forSources.remoteCacheJournal(String, ClientConfig, JournalInitialPosition, FunctionEx, PredicateEx)
which will pass onlyCREATED
andUPDATED
events and will project the event's key and new value into aMap.Entry
.static <T,
K, V> StreamSource<T> Sources.remoteCacheJournal
(String cacheName, ClientConfig clientConfig, JournalInitialPosition initialPos, FunctionEx<? super EventJournalCacheEvent<K, V>, ? extends T> projectionFn, PredicateEx<? super EventJournalCacheEvent<K, V>> predicateFn) Returns a source that will stream theEventJournalCacheEvent
events of the HazelcastICache
with the specified name from a remote cluster.static <K,
V> StreamSource<Map.Entry<K, V>> Sources.remoteMapJournal
(String mapName, ClientConfig clientConfig, JournalInitialPosition initialPos) Convenience forSources.remoteMapJournal(String, ClientConfig, JournalInitialPosition, FunctionEx, PredicateEx)
which will pass onlyADDED
andUPDATED
events and will project the event's key and new value into aMap.Entry
.static <T,
K, V> StreamSource<T> Sources.remoteMapJournal
(String mapName, ClientConfig clientConfig, JournalInitialPosition initialPos, FunctionEx<? super EventJournalMapEvent<K, V>, ? extends T> projectionFn, PredicateEx<? super EventJournalMapEvent<K, V>> predicateFn) Returns a source that will stream theEventJournalMapEvent
events of the HazelcastIMap
with the specified name from a remote cluster.static <K,
V> StreamSource<Map.Entry<K, V>> Sources.remoteMapJournal
(String mapName, DataConnectionRef dataConnectionRef, JournalInitialPosition initialPos) Convenience forSources.remoteMapJournal(String, DataConnectionRef, JournalInitialPosition, FunctionEx, PredicateEx)
which will pass onlyADDED
andUPDATED
events and will project the event's key and new value into aMap.Entry
.static <T,
K, V> StreamSource<T> Sources.remoteMapJournal
(String mapName, DataConnectionRef dataConnectionRef, JournalInitialPosition initialPos, FunctionEx<? super EventJournalMapEvent<K, V>, ? extends T> projectionFn, PredicateEx<? super EventJournalMapEvent<K, V>> predicateFn) The same as theSources.remoteMapJournal(String, ClientConfig, JournalInitialPosition, FunctionEx, PredicateEx)
method.StreamSource.setPartitionIdleTimeout
(long timeoutMillis) Sets a timeout after which idle partitions will be excluded from watermark coalescing.static StreamSource<String>
Convenience forsocket(host, port, charset)
with UTF-8 as the charset.static StreamSource<String>
Returns a source which connects to the specified socket and emits lines of text received from it.static <T> StreamSource<T>
Sources.streamFromProcessor
(String sourceName, ProcessorMetaSupplier metaSupplier) Returns an unbounded (event stream) source constructed directly from the given Core API processor meta-supplier.static <T> StreamSource<T>
Sources.streamFromProcessorWithWatermarks
(String sourceName, boolean supportsNativeTimestamps, FunctionEx<EventTimePolicy<? super T>, ProcessorMetaSupplier> metaSupplierFn) Returns an unbounded (event stream) source that will use the supplied function to create processor meta-suppliers as required by the Core API.Methods in com.hazelcast.jet.pipeline with parameters of type StreamSourceModifier and TypeMethodDescription<T> StreamSourceStage<T>
Pipeline.readFrom
(StreamSource<? extends T> source) Returns a pipeline stage that represents an unbounded data source (i.e., an event stream). -
Uses of StreamSource in com.hazelcast.jet.pipeline.test
Methods in com.hazelcast.jet.pipeline.test that return StreamSourceModifier and TypeMethodDescriptionstatic StreamSource<SimpleEvent>
TestSources.itemStream
(int itemsPerSecond) Returns a streaming source that generates events of typeSimpleEvent
at the specified rate.static <T> StreamSource<T>
TestSources.itemStream
(int itemsPerSecond, GeneratorFunction<? extends T> generatorFn) Returns a streaming source that generates events created by thegeneratorFn
at the specified rate.static StreamSource<Long>
TestSources.longStream
(long eventsPerSecond, long initialDelayMillis) Returns aStreamSource
that emits an ever-increasing sequence ofLong
numbers with native timestamps that are exactly the same amount of time apart, as specified by the suppliedeventsPerSecond
parameter.