public final class KafkaSources extends Object
|Modifier and Type||Method and Description|
Returns a source that consumes one or more Apache Kafka topics and emits items from them as
public static <K,V> Source<Map.Entry<K,V>> kafka(@Nonnull Properties properties, @Nonnull String... topics)
kafka(Properties, DistributedBiFunction, String...)wrapping the output in
public static <K,V,T> Source<T> kafka(@Nonnull Properties properties, @Nonnull DistributedBiFunction<K,V,T> projectionFn, @Nonnull String... topics)
The source creates a
KafkaConsumer for each
processor instance using the supplied
properties. It assigns a subset of Kafka partitions to each of
them using manual partition assignment (it ignores the
If snapshotting is enabled, partition offsets are saved to the snapshot. After restart, the source emits the events from the same offset.
If snapshotting is disabled, the source commits the offsets to Kafka
commitSync(). Note however that offsets can be committed before the
event is fully processed.
The processor completes only in the case of an error or if the job is
cancelled. IO failures are generally handled by Kafka producer and
do not cause the processor to fail.
Kafka consumer also does not return from
poll(timeout) if the
cluster is down. If
snapshotting is enabled, entire job might be blocked. This is a known
issue of Kafka (KAFKA-1894).
Refer to Kafka documentation for details.
snapshotInterval + timeToRestart + normalEventLag. We plan to address this issue in a future release.
properties- consumer properties broker address and key/value deserializers
projectionFn- function to create output objects from key and value
topics- the list of topics
Copyright © 2017 Hazelcast, Inc.. All Rights Reserved.