Uses of Interface
com.hazelcast.jet.pipeline.BatchSource
Package
Description
Apache Avro file read/write support for Hazelcast Jet.
Contains sources and sinks for Elasticsearch 7
Apache Hadoop read/write support for Hazelcast Jet.
Contains sources and sinks for MongoDB.
The Pipeline API is Jet's high-level API to build and execute
distributed computation jobs.
This package offers the
FileSourceBuilder
which allows you to construct various kinds of Pipeline
sources that read from local or distributed files.This package contains various mock sources to help with pipeline testing
and development.
AWS S3 read/write support for Hazelcast Jet.
-
Uses of BatchSource in com.hazelcast.jet.avro
Modifier and TypeMethodDescriptionAvroSourceBuilder.build()
Convenience forAvroSourceBuilder.build(BiFunctionEx)
.<T> BatchSource<T>
AvroSourceBuilder.build
(BiFunctionEx<String, ? super D, T> mapOutputFn) Builds a custom Avro fileBatchSource
with supplied components and the output functionmapOutputFn
.static <D> BatchSource<D>
AvroSources.files
(String directory, BiFunctionEx<String, org.apache.avro.generic.GenericRecord, D> mapOutputFn) Convenience forAvroSources.filesBuilder(String, SupplierEx)
which reads all the files in the supplied directory as generic records and emits the results of transforming each generic record with the supplied mapping function.static <D> BatchSource<D>
Convenience forAvroSources.filesBuilder(String, Class)
which reads all the files in the supplied directory as specific records using supplieddatumClass
. -
Uses of BatchSource in com.hazelcast.jet.elastic
Modifier and TypeMethodDescriptionElasticSourceBuilder.build()
Build ElasticsearchBatchSource
with supplied parametersstatic BatchSource<String>
ElasticSources.elastic()
Creates a source which queries local instance of Elasticsearch for all documentsstatic <T> BatchSource<T>
ElasticSources.elastic
(FunctionEx<? super org.elasticsearch.search.SearchHit, T> mapToItemFn) Creates a source which queries local instance of Elasticsearch for all documents.static BatchSource<String>
ElasticSources.elastic
(SupplierEx<org.elasticsearch.client.RestClientBuilder> clientFn) Creates a source which queries Elasticsearch using client obtained fromRestClientBuilder
supplier function.static <T> BatchSource<T>
ElasticSources.elastic
(SupplierEx<org.elasticsearch.client.RestClientBuilder> clientFn, FunctionEx<? super org.elasticsearch.search.SearchHit, T> mapToItemFn) Creates a source which queries Elasticsearch using client obtained fromRestClientBuilder
supplier function.static <T> BatchSource<T>
ElasticSources.elastic
(SupplierEx<org.elasticsearch.client.RestClientBuilder> clientFn, SupplierEx<org.elasticsearch.action.search.SearchRequest> searchRequestFn, FunctionEx<? super org.elasticsearch.search.SearchHit, T> mapToItemFn) Creates a source which queries Elasticsearch using client obtained fromRestHighLevelClient
supplier. -
Uses of BatchSource in com.hazelcast.jet.hadoop
Modifier and TypeMethodDescriptionstatic <K,
V, E> BatchSource<E> HadoopSources.inputFormat
(ConsumerEx<org.apache.hadoop.conf.Configuration> configureFn, BiFunctionEx<K, V, E> projectionFn) Returns a source that reads records from Apache Hadoop HDFS and emits the results of transforming each record (a key-value pair) with the supplied projection function.static <K,
V> BatchSource<Map.Entry<K, V>> HadoopSources.inputFormat
(org.apache.hadoop.conf.Configuration jobConf) Convenience forHadoopSources.inputFormat(Configuration, BiFunctionEx)
withMap.Entry
as its output type.static <K,
V, E> BatchSource<E> HadoopSources.inputFormat
(org.apache.hadoop.conf.Configuration configuration, BiFunctionEx<K, V, E> projectionFn) Returns a source that reads records from Apache Hadoop HDFS and emits the results of transforming each record (a key-value pair) with the supplied projection function. -
Uses of BatchSource in com.hazelcast.jet.mongodb
Modifier and TypeMethodDescriptionstatic BatchSource<org.bson.Document>
MongoSources.batch
(DataConnectionRef dataConnectionRef, String database, String collection, org.bson.conversions.Bson filter, org.bson.conversions.Bson projection) Returns a MongoDB batch source which queries the collection using givenfilter
and applies the givenprojection
on the documents.static BatchSource<org.bson.Document>
MongoSources.batch
(String connectionString, String database, String collection, org.bson.conversions.Bson filter, org.bson.conversions.Bson projection) Returns a MongoDB batch source which queries the collection using givenfilter
and applies the givenprojection
on the documents.MongoSourceBuilder.Batch.build()
Creates and returns the MongoDBBatchSource
. -
Uses of BatchSource in com.hazelcast.jet.pipeline
Modifier and TypeMethodDescriptionstatic <T> BatchSource<T>
Sources.batchFromProcessor
(String sourceName, ProcessorMetaSupplier metaSupplier) Returns a bounded (batch) source constructed directly from the given Core API processor meta-supplier.FileSourceBuilder.build()
Deprecated.<T> BatchSource<T>
FileSourceBuilder.build
(BiFunctionEx<String, String, ? extends T> mapOutputFn) Deprecated.<T> BatchSource<T>
FileSourceBuilder.build
(FunctionEx<? super Path, ? extends Stream<T>> readFileFn) Deprecated.<N> BatchSource<N>
RemoteMapSourceBuilder.build()
Build the source using the parameters set in this builder.SourceBuilder.Batch.build()
Builds and returns the batch source.static <K,
V> BatchSource<Map.Entry<K, V>> Returns a source that fetches entries from a HazelcastICache
with the given name and emits them asMap.Entry
.static BatchSource<String>
A source to read all files in a directory in a batch way.static <T> BatchSource<T>
Sources.jdbc
(SupplierEx<? extends Connection> newConnectionFn, ToResultSetFunction resultSetFn, FunctionEx<? super ResultSet, ? extends T> createOutputFn) Returns a source which connects to the specified database using the givennewConnectionFn
, queries the database and creates a result set using the the givenresultSetFn
.static <T> BatchSource<T>
Sources.jdbc
(DataConnectionRef dataConnectionRef, ToResultSetFunction resultSetFn, FunctionEx<? super ResultSet, ? extends T> createOutputFn) Returns a source which connects to the specified database using the givendataConnectionRef
, queries the database and creates a result set using the givenresultSetFn
.static <T> BatchSource<T>
Sources.jdbc
(String connectionURL, String query, FunctionEx<? super ResultSet, ? extends T> createOutputFn) Convenience forSources.jdbc(SupplierEx, ToResultSetFunction, FunctionEx)
.static <T> BatchSource<T>
Sources.jdbc
(String connectionURL, String query, Properties properties, FunctionEx<? super ResultSet, ? extends T> createOutputFn) Same as @{linkSources.jdbc(String, String, FunctionEx)
}static BatchSource<Map<String,
Object>> Convenience forSources.json(String, Class)
which converts each JSON string to aMap
.static <T> BatchSource<T>
A source to read all files in a directory in a batch way.static <T> BatchSource<T>
Returns a source that emits items retrieved from a HazelcastIList
.static <T> BatchSource<T>
Returns a source that emits items retrieved from a HazelcastIList
.static <K,
V> BatchSource<Map.Entry<K, V>> Returns a source that fetches entries from the given HazelcastIMap
and emits them asMap.Entry
.static <T,
K, V> BatchSource<T> Sources.map
(IMap<? extends K, ? extends V> map, Predicate<K, V> predicate, Projection<? super Map.Entry<K, V>, ? extends T> projection) Returns a source that fetches entries from the given HazelcastIMap
.static <K,
V> BatchSource<Map.Entry<K, V>> Returns a source that fetches entries from a local HazelcastIMap
with the specified name and emits them asMap.Entry
.static <T,
K, V> BatchSource<T> Sources.map
(String mapName, Predicate<K, V> predicate, Projection<? super Map.Entry<K, V>, ? extends T> projection) Returns a source that fetches entries from a local HazelcastIMap
with the specified name.static <K,
V> BatchSource<Map.Entry<K, V>> Sources.remoteCache
(String cacheName, ClientConfig clientConfig) Returns a source that fetches entries from the HazelcastICache
with the specified name in a remote cluster identified by the suppliedClientConfig
and emits them asMap.Entry
.static <T> BatchSource<T>
Sources.remoteList
(String listName, ClientConfig clientConfig) Returns a source that emits items retrieved from a HazelcastIList
in a remote cluster identified by the suppliedClientConfig
.static <K,
V> BatchSource<Map.Entry<K, V>> Sources.remoteMap
(String mapName, ClientConfig clientConfig) Returns a source that fetches entries from the HazelcastIMap
with the specified name in a remote cluster identified by the suppliedClientConfig
and emits them asMap.Entry
.static <T,
K, V> BatchSource<T> Sources.remoteMap
(String mapName, ClientConfig clientConfig, Predicate<K, V> predicate, Projection<? super Map.Entry<K, V>, ? extends T> projection) Returns a source that fetches entries from a remote HazelcastIMap
with the specified name in a remote cluster identified by the suppliedClientConfig
.static <K,
V> BatchSource<Map.Entry<K, V>> Sources.remoteMap
(String mapName, DataConnectionRef dataConnectionRef) The same as theSources.remoteMap(String, ClientConfig, Predicate, Projection)
method.static <T,
K, V> BatchSource<T> Sources.remoteMap
(String mapName, DataConnectionRef dataConnectionRef, Predicate<K, V> predicate, Projection<? super Map.Entry<K, V>, ? extends T> projection) The same as theSources.remoteMap(String, ClientConfig, Predicate, Projection)
method.Modifier and TypeMethodDescription<T> BatchStage<T>
Pipeline.readFrom
(BatchSource<? extends T> source) Returns a pipeline stage that represents a bounded (batch) data source. -
Uses of BatchSource in com.hazelcast.jet.pipeline.file
Modifier and TypeMethodDescriptionFileSourceBuilder.build()
Builds aBatchSource
based on the current state of the builder. -
Uses of BatchSource in com.hazelcast.jet.pipeline.test
Modifier and TypeMethodDescriptionstatic <T> BatchSource<T>
Returns a batch source which iterates through the supplied iterable and then terminates.static <T> BatchSource<T>
TestSources.items
(T... items) Returns a batch source which iterates through the supplied items and then terminates.static <T> BatchSource<T>
TestSources.itemsDistributed
(Iterable<? extends T> items) Returns a batch source which iterates through the supplied iterable and then terminates.static <T> BatchSource<T>
TestSources.itemsDistributed
(T... items) Returns a batch source which iterates through the supplied items and then terminates. -
Uses of BatchSource in com.hazelcast.jet.s3
Modifier and TypeMethodDescriptionstatic BatchSource<String>
S3Sources.s3
(List<String> bucketNames, String prefix, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier) Convenience forS3Sources.s3(List, String, Charset, SupplierEx, BiFunctionEx)
.static <I,
T> BatchSource<T> S3Sources.s3
(List<String> bucketNames, String prefix, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier, FunctionEx<? super InputStream, ? extends Stream<I>> readFileFn, BiFunctionEx<String, ? super I, ? extends T> mapFn) Creates an AWS S3BatchSource
which lists all the objects in the bucket-list using givenprefix
, reads them using providedreadFileFn
, transforms each read item to the desired output object using givenmapFn
and emits them to downstream.static <I,
T> BatchSource<T> S3Sources.s3
(List<String> bucketNames, String prefix, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier, TriFunction<? super InputStream, String, String, ? extends Stream<I>> readFileFn, BiFunctionEx<String, ? super I, ? extends T> mapFn) Creates an AWS S3BatchSource
which lists all the objects in the bucket-list using givenprefix
, reads them using providedreadFileFn
, transforms each read item to the desired output object using givenmapFn
and emits them to downstream.static <T> BatchSource<T>
S3Sources.s3
(List<String> bucketNames, String prefix, Charset charset, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier, BiFunctionEx<String, String, ? extends T> mapFn) Creates an AWS S3BatchSource
which lists all the objects in the bucket-list using givenprefix
, reads them line by line, transforms each line to the desired output object using givenmapFn
and emits them to downstream.
FileSources.files(java.lang.String)
.