com.hazelcast.jet.kafka

Class KafkaDataConnection

java.lang.Object

com.hazelcast.dataconnection.DataConnectionBase

com.hazelcast.jet.kafka.KafkaDataConnection

All Implemented Interfaces:

DataConnection

@Beta
public class KafkaDataConnection
extends DataConnectionBase

DataConnection implementation for Kafka.

KafkaDataConnection is usable both for sources and sinks.

Sources obtain KafkaConsumer instances using newConsumer(). Because the KafkaConsumer is not thread-safe so the DataConnection is used only to keep the connection metadata. A new instance is returned each time this method is called.

Sinks obtain KafkaProducer instances using getProducer(String). The producer instance may be either shared or single-use depending on the DataConnectionConfig.isShared() setting. Shared producer instance is limited to use with either no processing guarantees or at-least-once processing guarantees. Use with exactly-once is not possible.

The properties must be configured for both consumers and producers.

Since:

5.3

Constructor Summary

Constructors

Constructor and Description
KafkaDataConnection(DataConnectionConfig config) Create KafkaDataConnection based on given config

Method Summary

Modifier and Type	Method and Description
void	destroy() Called by the member when shutting down.
<K,V> org.apache.kafka.clients.producer.KafkaProducer<K,V>	getProducer(String transactionalId) Returns an instance of KafkaProducer based on the DataConnection configuration.
<K,V> org.apache.kafka.clients.producer.KafkaProducer<K,V>	getProducer(String transactionalId, Properties properties) Returns an instance of KafkaProducer based on the DataConnection configuration and given properties.
Collection<DataConnectionResource>	listResources() Returns list of DataConnectionResources accessible via this DataConnection.
<K,V> org.apache.kafka.clients.consumer.Consumer<K,V>	newConsumer() Creates new instance of KafkaConsumer based on the DataConnection configuration.
<K,V> org.apache.kafka.clients.consumer.Consumer<K,V>	newConsumer(Properties properties) Creates new instance of KafkaConsumer based on the DataConnection configuration and given properties parameter.
Collection<String>	resourceTypes() Returns the list of possible values for DataConnectionResource.type(), that will be returned when DataConnection.listResources() is called.

Methods inherited from class com.hazelcast.dataconnection.DataConnectionBase

getConfig, getName, release, retain

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.hazelcast.dataconnection.DataConnection

options

Constructor Detail

KafkaDataConnection

public KafkaDataConnection(@Nonnull
                           DataConnectionConfig config)

Create KafkaDataConnection based on given config

Method Detail

listResources

@Nonnull
public Collection<DataConnectionResource> listResources()

Description copied from interface: DataConnection

Returns list of DataConnectionResources accessible via this DataConnection.

It is not strictly required that the data connection lists all resources; a resource can be used even if it is not listed. For example, the list of resources in Oracle database might not include tables available through a database link. In fact, it might list no resources at all, perhaps if the security in the target system prevents reading of such a list.

The returned list contains up-to-date list of resources. Any changes (added or removed resources) must be reflected in subsequent calls to this method.

resourceTypes

@Nonnull
public Collection<String> resourceTypes()

Description copied from interface: DataConnection

Returns the list of possible values for DataConnectionResource.type(), that will be returned when DataConnection.listResources() is called. Returned values are case-insensitive, e.g. DataConnectionResource.type() may return MY_RES and this method my_res.

newConsumer

@Nonnull
public <K,V> org.apache.kafka.clients.consumer.Consumer<K,V> newConsumer()

Creates new instance of KafkaConsumer based on the DataConnection configuration. Always creates a new instance of the consumer because KafkaConsumer is not thread-safe. The caller is responsible for closing the consumer instance.

Returns:

consumer instance

newConsumer

@Nonnull
public <K,V> org.apache.kafka.clients.consumer.Consumer<K,V> newConsumer(Properties properties)

Creates new instance of KafkaConsumer based on the DataConnection configuration and given properties parameter. Always creates a new instance of the consumer because KafkaConsumer is not thread-safe. The caller is responsible for closing the consumer instance.

Parameters:

properties - mapping properties to merge with data connection options.

Returns:

consumer instance

getProducer

@Nonnull
public <K,V> org.apache.kafka.clients.producer.KafkaProducer<K,V> getProducer(@Nullable
                                                                                       String transactionalId)

Returns an instance of KafkaProducer based on the DataConnection configuration.

The caller is responsible for closing the producer instance. For non-shared producers the producer will be closed immediately upon calling close. For shared producers the producer will be closed when all users close the returned instance and this DataConnection is released by calling DataConnectionBase.release()

Parameters:

transactionalId - transaction id to pass as 'transactional.id' property to a new KafkaProducer instance, must be null for shared producer

getProducer

@Nonnull
public <K,V> org.apache.kafka.clients.producer.KafkaProducer<K,V> getProducer(@Nullable
                                                                                       String transactionalId,
                                                                                       @Nonnull
                                                                                       Properties properties)

Returns an instance of KafkaProducer based on the DataConnection configuration and given properties.

The caller is responsible for closing the producer instance. For non-shared producers the producer will be closed immediately upon calling close. For shared producers the producer will be closed when all users close the returned instance and this DataConnection is released by calling DataConnectionBase.release()

Parameters:

transactionalId - transaction id to pass as 'transactional.id' property to a new KafkaProducer instance, must be null for shared producer

properties - properties. E.g, SQL mappings provide separate options, and they should be merged with data connection's properties. These properties have higher priority than data connection properties. If KafkaDataConnection is shared, then HazelcastException would be thrown.

destroy

public void destroy()

Description copied from interface: DataConnection

Called by the member when shutting down. Should unconditionally close all connections and release resources.