Package | Description |
---|---|
com.hazelcast.connector |
Hazelcast 3 connector provides sources, sinks and enrichment using data
in Hazelcast 3 remote cluster
|
com.hazelcast.jet.avro |
Apache Avro file read/write support for Hazelcast Jet.
|
com.hazelcast.jet.elastic |
Contains sources and sinks for Elasticsearch 7
|
com.hazelcast.jet.hadoop |
Apache Hadoop read/write support for Hazelcast Jet.
|
com.hazelcast.jet.pipeline |
The Pipeline API is Jet's high-level API to build and execute
distributed computation jobs.
|
com.hazelcast.jet.pipeline.file |
This package offers the
FileSourceBuilder
which allows you to construct various kinds of Pipeline sources that read from local or distributed files. |
com.hazelcast.jet.pipeline.test |
This package contains various mock sources to help with pipeline testing
and development.
|
com.hazelcast.jet.s3 |
AWS S3 read/write support for Hazelcast Jet.
|
Modifier and Type | Method and Description |
---|---|
static <K,V> BatchSource<Map.Entry<K,V>> |
Hz3Sources.remoteMap(String mapName,
String clientXml)
Returns a source that fetches entries from the Hazelcast
IMap
with the specified name in a remote cluster identified by the supplied
XML configuration. |
Modifier and Type | Method and Description |
---|---|
BatchSource<D> |
AvroSourceBuilder.build()
Convenience for
AvroSourceBuilder.build(BiFunctionEx) . |
<T> BatchSource<T> |
AvroSourceBuilder.build(BiFunctionEx<String,? super D,T> mapOutputFn)
Builds a custom Avro file
BatchSource with supplied components
and the output function mapOutputFn . |
static <D> BatchSource<D> |
AvroSources.files(String directory,
BiFunctionEx<String,org.apache.avro.generic.GenericRecord,D> mapOutputFn)
Convenience for
AvroSources.filesBuilder(String, SupplierEx) which
reads all the files in the supplied directory as generic records and
emits the results of transforming each generic record with the supplied
mapping function. |
static <D> BatchSource<D> |
AvroSources.files(String directory,
Class<D> datumClass)
Convenience for
AvroSources.filesBuilder(String, Class) which
reads all the files in the supplied directory as specific records using
supplied datumClass . |
Modifier and Type | Method and Description |
---|---|
BatchSource<T> |
ElasticSourceBuilder.build()
Build Elasticsearch
BatchSource with supplied parameters |
static BatchSource<String> |
ElasticSources.elastic()
Creates a source which queries local instance of Elasticsearch for all
documents
|
static <T> BatchSource<T> |
ElasticSources.elastic(FunctionEx<? super org.elasticsearch.search.SearchHit,T> mapToItemFn)
Creates a source which queries local instance of Elasticsearch for all
documents.
|
static BatchSource<String> |
ElasticSources.elastic(SupplierEx<org.elasticsearch.client.RestClientBuilder> clientFn)
Creates a source which queries Elasticsearch using client obtained from
RestClientBuilder supplier function. |
static <T> BatchSource<T> |
ElasticSources.elastic(SupplierEx<org.elasticsearch.client.RestClientBuilder> clientFn,
FunctionEx<? super org.elasticsearch.search.SearchHit,T> mapToItemFn)
Creates a source which queries Elasticsearch using client obtained from
RestClientBuilder supplier function. |
static <T> BatchSource<T> |
ElasticSources.elastic(SupplierEx<org.elasticsearch.client.RestClientBuilder> clientFn,
SupplierEx<org.elasticsearch.action.search.SearchRequest> searchRequestFn,
FunctionEx<? super org.elasticsearch.search.SearchHit,T> mapToItemFn)
Creates a source which queries Elasticsearch using client obtained from
RestHighLevelClient supplier. |
Modifier and Type | Method and Description |
---|---|
static <K,V> BatchSource<Map.Entry<K,V>> |
HadoopSources.inputFormat(org.apache.hadoop.conf.Configuration jobConf)
Convenience for
HadoopSources.inputFormat(Configuration, BiFunctionEx)
with Map.Entry as its output type. |
static <K,V,E> BatchSource<E> |
HadoopSources.inputFormat(org.apache.hadoop.conf.Configuration configuration,
BiFunctionEx<K,V,E> projectionFn)
Returns a source that reads records from Apache Hadoop HDFS and emits
the results of transforming each record (a key-value pair) with the
supplied projection function.
|
static <K,V,E> BatchSource<E> |
HadoopSources.inputFormat(ConsumerEx<org.apache.hadoop.conf.Configuration> configureFn,
BiFunctionEx<K,V,E> projectionFn)
Returns a source that reads records from Apache Hadoop HDFS and emits
the results of transforming each record (a key-value pair) with the
supplied projection function.
|
Modifier and Type | Method and Description |
---|---|
static <T> BatchSource<T> |
Sources.batchFromProcessor(String sourceName,
ProcessorMetaSupplier metaSupplier)
Returns a bounded (batch) source constructed directly from the given
Core API processor meta-supplier.
|
BatchSource<String> |
FileSourceBuilder.build()
Deprecated.
Use
FileSources.files(java.lang.String) . Will be removed in Jet 5.0. |
BatchSource<T> |
SourceBuilder.Batch.build()
Builds and returns the batch source.
|
<T> BatchSource<T> |
FileSourceBuilder.build(BiFunctionEx<String,String,? extends T> mapOutputFn)
Deprecated.
Use
FileSources.files(java.lang.String) . Will be removed in Jet 5.0. |
<T> BatchSource<T> |
FileSourceBuilder.build(FunctionEx<? super Path,? extends Stream<T>> readFileFn)
Deprecated.
Use
FileSources.files(java.lang.String) . Will be removed in Jet 5.0. |
static <K,V> BatchSource<Map.Entry<K,V>> |
Sources.cache(String cacheName)
Returns a source that fetches entries from a Hazelcast
ICache
with the given name and emits them as Map.Entry . |
static BatchSource<String> |
Sources.files(String directory)
A source to read all files in a directory in a batch way.
|
static <T> BatchSource<T> |
Sources.jdbc(String connectionURL,
String query,
FunctionEx<? super ResultSet,? extends T> createOutputFn)
Convenience for
Sources.jdbc(SupplierEx,
ToResultSetFunction, FunctionEx) . |
static <T> BatchSource<T> |
Sources.jdbc(SupplierEx<? extends Connection> newConnectionFn,
ToResultSetFunction resultSetFn,
FunctionEx<? super ResultSet,? extends T> createOutputFn)
Returns a source which connects to the specified database using the given
newConnectionFn , queries the database and creates a result set
using the the given resultSetFn . |
static BatchSource<Map<String,Object>> |
Sources.json(String directory)
Convenience for
Sources.json(String, Class) which converts each
JSON string to a Map . |
static <T> BatchSource<T> |
Sources.json(String directory,
Class<T> type)
A source to read all files in a directory in a batch way.
|
static <T> BatchSource<T> |
Sources.list(IList<? extends T> list)
Returns a source that emits items retrieved from a Hazelcast
IList . |
static <T> BatchSource<T> |
Sources.list(String listName)
Returns a source that emits items retrieved from a Hazelcast
IList . |
static <K,V> BatchSource<Map.Entry<K,V>> |
Sources.map(IMap<? extends K,? extends V> map)
Returns a source that fetches entries from the given Hazelcast
IMap and emits them as Map.Entry . |
static <T,K,V> BatchSource<T> |
Sources.map(IMap<? extends K,? extends V> map,
Predicate<K,V> predicate,
Projection<? super Map.Entry<K,V>,? extends T> projection)
Returns a source that fetches entries from the given Hazelcast
IMap . |
static <K,V> BatchSource<Map.Entry<K,V>> |
Sources.map(String mapName)
Returns a source that fetches entries from a local Hazelcast
IMap
with the specified name and emits them as Map.Entry . |
static <T,K,V> BatchSource<T> |
Sources.map(String mapName,
Predicate<K,V> predicate,
Projection<? super Map.Entry<K,V>,? extends T> projection)
Returns a source that fetches entries from a local Hazelcast
IMap with the specified name. |
static <K,V> BatchSource<Map.Entry<K,V>> |
Sources.remoteCache(String cacheName,
ClientConfig clientConfig)
Returns a source that fetches entries from the Hazelcast
ICache
with the specified name in a remote cluster identified by the supplied
ClientConfig and emits them as Map.Entry . |
static <T> BatchSource<T> |
Sources.remoteList(String listName,
ClientConfig clientConfig)
Returns a source that emits items retrieved from a Hazelcast
IList in a remote cluster identified by the supplied ClientConfig . |
static <K,V> BatchSource<Map.Entry<K,V>> |
Sources.remoteMap(String mapName,
ClientConfig clientConfig)
Returns a source that fetches entries from the Hazelcast
IMap
with the specified name in a remote cluster identified by the supplied
ClientConfig and emits them as Map.Entry . |
static <T,K,V> BatchSource<T> |
Sources.remoteMap(String mapName,
ClientConfig clientConfig,
Predicate<K,V> predicate,
Projection<? super Map.Entry<K,V>,? extends T> projection)
Returns a source that fetches entries from a remote Hazelcast
IMap with the specified name in a remote cluster identified by the
supplied ClientConfig . |
Modifier and Type | Method and Description |
---|---|
<T> BatchStage<T> |
Pipeline.readFrom(BatchSource<? extends T> source)
Returns a pipeline stage that represents a bounded (batch) data source.
|
Modifier and Type | Method and Description |
---|---|
BatchSource<T> |
FileSourceBuilder.build()
Builds a
BatchSource based on the current state of the builder. |
Modifier and Type | Method and Description |
---|---|
static <T> BatchSource<T> |
TestSources.items(Iterable<? extends T> items)
Returns a batch source which iterates through the supplied iterable and
then terminates.
|
static <T> BatchSource<T> |
TestSources.items(T... items)
Returns a batch source which iterates through the supplied items and
then terminates.
|
static <T> BatchSource<T> |
TestSources.itemsDistributed(Iterable<? extends T> items)
Returns a batch source which iterates through the supplied iterable and
then terminates.
|
static <T> BatchSource<T> |
TestSources.itemsDistributed(T... items)
Returns a batch source which iterates through the supplied items and
then terminates.
|
Modifier and Type | Method and Description |
---|---|
static <T> BatchSource<T> |
S3Sources.s3(List<String> bucketNames,
String prefix,
Charset charset,
SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier,
BiFunctionEx<String,String,? extends T> mapFn)
Creates an AWS S3
BatchSource which lists all the objects in the
bucket-list using given prefix , reads them line by line,
transforms each line to the desired output object using given mapFn and emits them to downstream. |
static BatchSource<String> |
S3Sources.s3(List<String> bucketNames,
String prefix,
SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier)
Convenience for
S3Sources.s3(List, String, Charset, SupplierEx, BiFunctionEx) . |
static <I,T> BatchSource<T> |
S3Sources.s3(List<String> bucketNames,
String prefix,
SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier,
FunctionEx<? super InputStream,? extends Stream<I>> readFileFn,
BiFunctionEx<String,? super I,? extends T> mapFn)
Creates an AWS S3
BatchSource which lists all the objects in the
bucket-list using given prefix , reads them using provided readFileFn , transforms each read item to the desired output object
using given mapFn and emits them to downstream. |
static <I,T> BatchSource<T> |
S3Sources.s3(List<String> bucketNames,
String prefix,
SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier,
TriFunction<? super InputStream,String,String,? extends Stream<I>> readFileFn,
BiFunctionEx<String,? super I,? extends T> mapFn)
Creates an AWS S3
BatchSource which lists all the objects in the
bucket-list using given prefix , reads them using provided readFileFn , transforms each read item to the desired output object
using given mapFn and emits them to downstream. |
Copyright © 2022 Hazelcast, Inc.. All rights reserved.