Package com.hazelcast.jet.avro
Class AvroSources
java.lang.Object
com.hazelcast.jet.avro.AvroSources
Contains factory methods for Apache Avro sources.
- Since:
- Jet 3.0
-
Method Summary
Modifier and TypeMethodDescriptionstatic <D> BatchSource<D>
files
(String directory, BiFunctionEx<String, org.apache.avro.generic.GenericRecord, D> mapOutputFn) Convenience forfilesBuilder(String, SupplierEx)
which reads all the files in the supplied directory as generic records and emits the results of transforming each generic record with the supplied mapping function.static <D> BatchSource<D>
Convenience forfilesBuilder(String, Class)
which reads all the files in the supplied directory as specific records using supplieddatumClass
.static <D> AvroSourceBuilder<D>
filesBuilder
(String directory, SupplierEx<? extends org.apache.avro.io.DatumReader<D>> datumReaderSupplier) Returns a builder object that offers a step-by-step fluent API to build a custom Avro file source for the Pipeline API.static <D> AvroSourceBuilder<D>
filesBuilder
(String directory, Class<D> recordClass) Returns a builder object that offers a step-by-step fluent API to build a custom Avro file source for the Pipeline API.
-
Method Details
-
filesBuilder
@Nonnull public static <D> AvroSourceBuilder<D> filesBuilder(@Nonnull String directory, @Nonnull Class<D> recordClass) Returns a builder object that offers a step-by-step fluent API to build a custom Avro file source for the Pipeline API. The source reads records from Apache Avro files in a directory (but not its subdirectories).- Type Parameters:
D
- the type of the datum- Parameters:
directory
- parent directory of the filesrecordClass
- the class to read
-
filesBuilder
@Nonnull public static <D> AvroSourceBuilder<D> filesBuilder(@Nonnull String directory, @Nonnull SupplierEx<? extends org.apache.avro.io.DatumReader<D>> datumReaderSupplier) Returns a builder object that offers a step-by-step fluent API to build a custom Avro file source for the Pipeline API. The source reads records from Apache Avro files in a directory (but not its subdirectories).- Type Parameters:
D
- the type of the datum- Parameters:
directory
- parent directory of the filesdatumReaderSupplier
- the supplier of datum reader which reads records from the files
-
files
@Nonnull public static <D> BatchSource<D> files(@Nonnull String directory, @Nonnull Class<D> datumClass) Convenience forfilesBuilder(String, Class)
which reads all the files in the supplied directory as specific records using supplieddatumClass
. IfdatumClass
implementsSpecificRecord
,SpecificDatumReader
is used to read the records,ReflectDatumReader
is used otherwise. -
files
@Nonnull public static <D> BatchSource<D> files(@Nonnull String directory, @Nonnull BiFunctionEx<String, org.apache.avro.generic.GenericRecord, D> mapOutputFn) Convenience forfilesBuilder(String, SupplierEx)
which reads all the files in the supplied directory as generic records and emits the results of transforming each generic record with the supplied mapping function.
-