com.hazelcast.jet.s3

Class S3Sinks

java.lang.Object

com.hazelcast.jet.s3.S3Sinks

public final class S3Sinks
extends Object

Contains factory methods for creating AWS S3 sinks.

Method Summary

Modifier and Type	Method and Description
static <T> Sink<? super T>	s3(String bucketName, String prefix, Charset charset, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier, FunctionEx<? super T,String> toStringFn) Creates an AWS S3 Sink which writes items to files into the given bucket.
static <T> Sink<? super T>	s3(String bucketName, SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier) Convenience for s3(String, String, Charset, SupplierEx, FunctionEx) Uses Object.toString() to convert the items to lines.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

s3

@Nonnull
public static <T> Sink<? super T> s3(@Nonnull
                                              String bucketName,
                                              @Nonnull
                                              SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier)

Convenience for s3(String, String, Charset, SupplierEx, FunctionEx) Uses Object.toString() to convert the items to lines.

s3

@Nonnull
public static <T> Sink<? super T> s3(@Nonnull
                                              String bucketName,
                                              @Nullable
                                              String prefix,
                                              @Nonnull
                                              Charset charset,
                                              @Nonnull
                                              SupplierEx<? extends software.amazon.awssdk.services.s3.S3Client> clientSupplier,
                                              @Nonnull
                                              FunctionEx<? super T,String> toStringFn)

Creates an AWS S3 Sink which writes items to files into the given bucket. Sink converts each item to string using given toStringFn and writes it as a line. The sink creates a file in the bucket for each processor instance. Name of the file will include an user provided prefix (if defined) and processor's global index, for example the processor having the index 2 with prefix my-object- will create the object my-object-2.

No state is saved to snapshot for this sink. If the job is restarted previously written files will be overwritten.

The default local parallelism for this sink is 1.

Here is an example which reads from a map and writes the entries to given bucket using Object.toString() to convert the values to a line.

 Pipeline p = Pipeline.create();
 p.readFrom(Sources.map("map"))
  .writeTo(S3Sinks.s3("bucket", "my-map-", StandardCharsets.UTF_8,
      () -> S3Client.create(),
      Object::toString
 ));
 

Type Parameters:

T - type of the items the sink accepts

Parameters:

bucketName - the name of the bucket

prefix - the prefix to be included in the file name

charset - the charset to be used when encoding the strings

clientSupplier - S3 client supplier

toStringFn - the function which converts each item to its string representation