com.hazelcast.jet.kafka

Class KafkaSinks

java.lang.Object

com.hazelcast.jet.kafka.KafkaSinks

public final class KafkaSinks
extends Object

Contains factory methods for Apache Kafka sinks.

Method Summary

Modifier and Type	Method and Description
static <E,K,V> Sink<E>	kafka(Properties properties, DistributedFunction<? super E,org.apache.kafka.clients.producer.ProducerRecord<K,V>> toRecordFn) Returns a source that publishes messages to an Apache Kafka topic.
static <K,V> Sink<Map.Entry<K,V>>	kafka(Properties properties, String topic) Convenience for kafka(Properties, String, DistributedFunction, DistributedFunction) which expects Map.Entry<K, V> as input and extracts its key and value parts to be published to Kafka.
static <E,K,V> Sink<E>	kafka(Properties properties, String topic, DistributedFunction<? super E,K> extractKeyFn, DistributedFunction<? super E,V> extractValueFn) Convenience for kafka(Properties, DistributedFunction) which creates a ProducerRecord using the given topic and the given key and value mapping functions

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

kafka

@Nonnull
public static <E,K,V> Sink<E> kafka(@Nonnull
                                             Properties properties,
                                             @Nonnull
                                             DistributedFunction<? super E,org.apache.kafka.clients.producer.ProducerRecord<K,V>> toRecordFn)

Returns a source that publishes messages to an Apache Kafka topic. It transforms each received item to a ProducerRecord using the supplied mapping function.

The source creates a single KafkaProducer per cluster member using the supplied properties.

Behavior on job restart: the processor is stateless. On snapshot we only make sure that all async operations are done. If the job is restarted, duplicate events can occur. If you need exactly-once behavior, you must ensure idempotence on the application level.

IO failures are generally handled by Kafka producer and do not cause the processor to fail. Refer to Kafka documentation for details.

Default local parallelism for this processor is 2 (or less if less CPUs are available).

Type Parameters:

E - type of stream item

K - type of the key published to Kafka

V - type of the value published to Kafka

Parameters:

properties - producer properties which should contain broker address and key/value serializers

toRecordFn - function that extracts the key from the stream item

kafka

@Nonnull
public static <E,K,V> Sink<E> kafka(@Nonnull
                                             Properties properties,
                                             @Nonnull
                                             String topic,
                                             @Nonnull
                                             DistributedFunction<? super E,K> extractKeyFn,
                                             @Nonnull
                                             DistributedFunction<? super E,V> extractValueFn)

Convenience for kafka(Properties, DistributedFunction) which creates a ProducerRecord using the given topic and the given key and value mapping functions

Type Parameters:

E - type of stream item

K - type of the key published to Kafka

V - type of the value published to Kafka

Parameters:

properties - producer properties which should contain broker address and key/value serializers

topic - name of the Kafka topic to publish to

extractKeyFn - function that extracts the key from the stream item

extractValueFn - function that extracts the value from the stream item

kafka

@Nonnull
public static <K,V> Sink<Map.Entry<K,V>> kafka(@Nonnull
                                                        Properties properties,
                                                        @Nonnull
                                                        String topic)

Convenience for kafka(Properties, String, DistributedFunction, DistributedFunction) which expects Map.Entry<K, V> as input and extracts its key and value parts to be published to Kafka.

Type Parameters:

K - type of the key published to Kafka

V - type of the value published to Kafka

Parameters:

properties - producer properties which should contain broker address and key/value serializers

topic - Kafka topic name to publish to