Class S3Sinks


public final class S3Sinks extends Object
Contains factory methods for creating AWS S3 sinks.
  • Method Details

    • s3

      @Nonnull public static <T> Sink<? super T> s3(@Nonnull String bucketName, @Nonnull SupplierEx<? extends> clientSupplier)
      Convenience for s3(String, String, Charset, SupplierEx, FunctionEx) Uses Object.toString() to convert the items to lines.
    • s3

      @Nonnull public static <T> Sink<? super T> s3(@Nonnull String bucketName, @Nullable String prefix, @Nonnull Charset charset, @Nonnull SupplierEx<? extends> clientSupplier, @Nonnull FunctionEx<? super T,String> toStringFn)
      Creates an AWS S3 Sink which writes items to files into the given bucket. Sink converts each item to string using given toStringFn and writes it as a line. The sink creates a file in the bucket for each processor instance. Name of the file will include an user provided prefix (if defined) and processor's global index, for example the processor having the index 2 with prefix my-object- will create the object my-object-2.

      No state is saved to snapshot for this sink. If the job is restarted previously written files will be overwritten.

      The default local parallelism for this sink is 1.

      Here is an example which reads from a map and writes the entries to given bucket using Object.toString() to convert the values to a line.

       Pipeline p = Pipeline.create();
        .writeTo(S3Sinks.s3("bucket", "my-map-", StandardCharsets.UTF_8,
            () -> S3Client.create(),
      Type Parameters:
      T - type of the items the sink accepts
      bucketName - the name of the bucket
      prefix - the prefix to be included in the file name
      charset - the charset to be used when encoding the strings
      clientSupplier - S3 client supplier
      toStringFn - the function which converts each item to its string representation