com.hazelcast.jet.kinesis

Class KinesisSources.Builder<T>

java.lang.Object

com.hazelcast.jet.kinesis.KinesisSources.Builder<T>

Type Parameters:

T - Result type returned from the source.

Enclosing class:

KinesisSources

public static final class KinesisSources.Builder<T>
extends Object

Fluent builder for constructing the Kinesis source and setting its configuration parameters.

Method Summary

Modifier and Type	Method and Description
StreamSource<T>	build() Constructs the source based on the options provided so far.
KinesisSources.Builder<T>	withCredentials(String accessKey, String secretKey) Specifies the AWS credentials to use for authentication purposes.
KinesisSources.Builder<T>	withEndpoint(String endpoint) Specifies the AWS Kinesis endpoint (URL of the entry point for the AWS web service) to connect to.
KinesisSources.Builder<T>	withInitialShardIteratorRule(String shardIdRegExp, String shardIteratorType, String parameter) Specifies how initial reads of the shards should be done (see shardIteratorType for available options).
<T_NEW> KinesisSources.Builder<T_NEW>	withProjectionFn(BiFunctionEx<com.amazonaws.services.kinesis.model.Record,com.amazonaws.services.kinesis.model.Shard,T_NEW> projectionFn) Specifies projection function, that will map input Record and Shard from which this record was read into user-defined type.
KinesisSources.Builder<T>	withRegion(String region) Specifies the AWS Region (collection of AWS resources in a geographic area) to connect to.
KinesisSources.Builder<T>	withRetryStrategy(RetryStrategy retryStrategy) Specifies how the source should behave when reading data from the stream fails.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

withEndpoint

@Nonnull
public KinesisSources.Builder<T> withEndpoint(@Nullable
                                                       String endpoint)

Specifies the AWS Kinesis endpoint (URL of the entry point for the AWS web service) to connect to. The general syntax of these endpoint URLs is "protocol://service-code.region-code.amazonaws.com", so for example, for Kinesis, for the us-west-2 region, we could have "https://dynamodb.us-west-2.amazonaws.com". For local testing, it might be "http://localhost:4566".

If not specified (or specified as null), the default endpoint for the specified region will be used.

withRegion

@Nonnull
public KinesisSources.Builder<T> withRegion(@Nullable
                                                     String region)

Specifies the AWS Region (collection of AWS resources in a geographic area) to connect to. Region names are of form "us-west-1", "eu-central-1" and so on.

If not specified (or specified as null), the default region set via external means will be used (either from your local .aws/config file or the AWS_REGION environment variable). If no such default is set, then "us-east-1" will be used.

withCredentials

@Nonnull
public KinesisSources.Builder<T> withCredentials(@Nullable
                                                          String accessKey,
                                                          @Nullable
                                                          String secretKey)

Specifies the AWS credentials to use for authentication purposes.

If not specified (or specified as null), then keys specified via external means will be used. This can mean the local .aws/credentials file or the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environmental variables.

Either both keys must be set to non-null values or neither.

withRetryStrategy

@Nonnull
public KinesisSources.Builder<T> withRetryStrategy(@Nonnull
                                                            RetryStrategy retryStrategy)

Specifies how the source should behave when reading data from the stream fails. The default behavior retries the read operation indefinitely, but after an exponentially increasing delay, it starts with 100 milliseconds and doubles on each subsequent failure. A successful read resets it. The delay is capped at 3 seconds.

withInitialShardIteratorRule

@Nonnull
public KinesisSources.Builder<T> withInitialShardIteratorRule(@Nonnull
                                                                       String shardIdRegExp,
                                                                       @Nonnull
                                                                       String shardIteratorType,
                                                                       @Nullable
                                                                       String parameter)

Specifies how initial reads of the shards should be done (see shardIteratorType for available options). Initial read means the moment when a pipeline initiates reading a shard for the first time. If a pipeline's execution is being resumed based on a snapshot and there is a saved read offset for the shard then it is NOT considered an initial read.

Each call of this method registers one rule which applies to any number of shards, depending on the shardIdRegExp matching the shard's id or not. The syntax of regular expressions is as defined by java.util.regex.Pattern. The method can be called any number of times, the rules will be evaluated in the order they have been registered. Only the first rule that matches a shard will be applied for that shard.

Type of rules allowed, together with their meaning, are:

• AT_SEQUENCE_NUMBER: start reading the shard at the record with the specified sequence number
• AFTER_SEQUENCE_NUMBER: start reading the shard right after the record with the specified sequence number
• AT_TIMESTAMP: start reading at the first record with a timestamp equal to or greater than the provided one
• TRIM_HORIZON: start reading at the oldest record in the shard, ie. read all available records
• LATEST: start reading just after the most recent record in the shard, ie. read only records that will be ingested later into the shard

Depending on the rule specified the optional parameter must also be specified. For AT_TIMESTAMP it needs to be a timestamp in the Unix epoch date format, with millisecond precision. For AT_SEQUENCE_NUMBER and AFTER_SEQUENCE_NUMBER it must be a sequence number accepted by KDS. For other types it must be left null.

NOTE: The AT_TIMESTAMP version, even though supported by Kinesis, doesn't currently work properly. This is due to be a bug in AWS SDK v1, used by the source. AWS SDK v2 fixes the problem and hopefully this source will be switched to using that in future Jet releases.

withProjectionFn

@Nonnull
public <T_NEW> KinesisSources.Builder<T_NEW> withProjectionFn(@Nonnull
                                                                       BiFunctionEx<com.amazonaws.services.kinesis.model.Record,com.amazonaws.services.kinesis.model.Shard,T_NEW> projectionFn)

Specifies projection function, that will map input Record and Shard from which this record was read into user-defined type. If not provided, source will return Map.Entry<String, byte[]> with Record.getPartitionKey() as key and Record.getData() as value.

build

@Nonnull
public StreamSource<T> build()

Constructs the source based on the options provided so far.