com.hazelcast.jet.kafka

Class KafkaDataLink

java.lang.Object

com.hazelcast.datalink.DataLinkBase

com.hazelcast.jet.kafka.KafkaDataLink

All Implemented Interfaces:

DataLink

@Beta
public class KafkaDataLink
extends DataLinkBase
implements DataLink

DataLink implementation for Kafka.

KafkaDataLink is usable both for sources and sinks.

Sources obtain KafkaConsumer instances using newConsumer(). Because the KafkaConsumer is not thread-safe so the DataLink is used only to keep the connection metadata. A new instance is returned each time this method is called.

Sinks obtain KafkaProducer instances using getProducer(String). The producer instance may be either shared or not shared depending on the DataLinkConfig.isShared() setting. Shared producer instance is limited to use with either no processing guarantees or at-least-once processing guarantees. Use with exactly-once is not possible.

The properties must be configured for both consumers and producers.

Since:

5.3

Constructor Summary

Constructors

Constructor and Description
KafkaDataLink(DataLinkConfig config) Create KafkaDataLink based on given config

Method Summary

Modifier and Type	Method and Description
void	destroy() Called by the member when shutting down.
<K,V> org.apache.kafka.clients.producer.KafkaProducer<K,V>	getProducer(String transactionalId) Returns an instance of KafkaProducer based on the DataLink configuration.
Collection<DataLinkResource>	listResources() Returns list of DataLinkResources accessible via this DataLink.
<K,V> org.apache.kafka.clients.consumer.Consumer<K,V>	newConsumer() Creates new instance of KafkaConsumer based on the DataLink configuration.

Methods inherited from class com.hazelcast.datalink.DataLinkBase

getConfig, getName, release, retain

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.hazelcast.datalink.DataLink

getConfig, getName, options, release, retain

Constructor Detail

KafkaDataLink

public KafkaDataLink(@Nonnull
                     DataLinkConfig config)

Create KafkaDataLink based on given config

Method Detail

listResources

@Nonnull
public Collection<DataLinkResource> listResources()

Description copied from interface: DataLink

Returns list of DataLinkResources accessible via this DataLink.

It is not strictly required that the data link lists all resources; a resource can be used even if it is not listed. For example, the list of resources in Oracle database might not include tables available through a database link. In fact, it might list no resources at all, perhaps if the security in the target system prevents reading of such a list.

The returned list contains up-to-date list of resources. Any changes (added or removed resources) must be reflected in subsequent calls to this method.

Specified by:

listResources in interface DataLink

newConsumer

@Nonnull
public <K,V> org.apache.kafka.clients.consumer.Consumer<K,V> newConsumer()

Creates new instance of KafkaConsumer based on the DataLink configuration. Always creates a new instance of the consumer because KafkaConsumer is not thread-safe. The caller is responsible for closing the consumer instance.

Returns:

consumer instance

getProducer

@Nonnull
public <K,V> org.apache.kafka.clients.producer.KafkaProducer<K,V> getProducer(@Nullable
                                                                                       String transactionalId)

Returns an instance of KafkaProducer based on the DataLink configuration.

The caller is responsible for closing the producer instance. For non-shared producers the producer will be closed immediately upon calling close. For shared producers the producer will be closed when all users close the returned instance and this DataLink is released by calling DataLinkBase.release()

Parameters:

transactionalId - transaction id to pass as 'transactional.id' property to a new KafkaProducer instance, must be null for shared producer

destroy

public void destroy()

Description copied from interface: DataLink

Called by the member when shutting down. Should unconditionally close all connections and release resources.

Specified by:

destroy in interface DataLink