com.hazelcast.jet

Class KafkaSources

java.lang.Object

com.hazelcast.jet.KafkaSources

public final class KafkaSources
extends Object

Contains factory methods for Apache Kafka sources.

Method Summary

Modifier and Type	Method and Description
static <K,V,T> Source<T>	kafka(Properties properties, DistributedBiFunction<K,V,T> projectionFn, String... topics) Returns a source that consumes one or more Apache Kafka topics and emits items from them as Map.Entry instances.
static <K,V> Source<Map.Entry<K,V>>	kafka(Properties properties, String... topics) Convenience for kafka(Properties, DistributedBiFunction, String...) wrapping the output in Map.Entry.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

kafka

public static <K,V> Source<Map.Entry<K,V>> kafka(@Nonnull
                                                 Properties properties,
                                                 @Nonnull
                                                 String... topics)

Convenience for kafka(Properties, DistributedBiFunction, String...) wrapping the output in Map.Entry.

kafka

public static <K,V,T> Source<T> kafka(@Nonnull
                                      Properties properties,
                                      @Nonnull
                                      DistributedBiFunction<K,V,T> projectionFn,
                                      @Nonnull
                                      String... topics)

Returns a source that consumes one or more Apache Kafka topics and emits items from them as Map.Entry instances.

The source creates a KafkaConsumer for each processor instance using the supplied properties. It assigns a subset of Kafka partitions to each of them using manual partition assignment (it ignores the group.id property).

If snapshotting is enabled, partition offsets are saved to the snapshot. After restart, the source emits the events from the same offset.

If snapshotting is disabled, the source commits the offsets to Kafka using commitSync(). Note however that offsets can be committed before the event is fully processed.

The processor completes only in the case of an error or if the job is cancelled. IO failures are generally handled by Kafka producer and do not cause the processor to fail. Kafka consumer also does not return from poll(timeout) if the cluster is down. If snapshotting is enabled, entire job might be blocked. This is a known issue of Kafka (KAFKA-1894). Refer to Kafka documentation for details.

Issue when "catching up"

The processor reads partitions one by one: it gets events from one partition and then moves to the next one etc. This adds time disorder to events: it might emit very recent event from partition1 while not yet emitting an old event from partition2. If watermarks are added to the stream later, the allowed event lag should accommodate this disorder. Most notably, the "catching up" happens after the job is restarted, when events since the last snapshot are reprocessed in a burst. In order to not lose any events, the lag should be configured to at least snapshotInterval + timeToRestart + normalEventLag. We plan to address this issue in a future release.

Parameters:

properties - consumer properties broker address and key/value deserializers

projectionFn - function to create output objects from key and value

topics - the list of topics