Package com.hazelcast.jet.kafka

Class KafkaDataConnection

java.lang.Object
com.hazelcast.dataconnection.DataConnectionBase
com.hazelcast.jet.kafka.KafkaDataConnection
All Implemented Interfaces:
DataConnection
public class KafkaDataConnection extends DataConnectionBase
DataConnection implementation for Kafka.

KafkaDataConnection is usable both for sources and sinks.

Sources obtain KafkaConsumer instances using newConsumer(). Because the KafkaConsumer is not thread-safe so the DataConnection is used only to keep the connection metadata. A new instance is returned each time this method is called.

Sinks obtain KafkaProducer instances using getProducer(String). The producer instance may be either shared or single-use depending on the DataConnectionConfig.isShared() setting. Shared producer instance is limited to use with either no processing guarantees or at-least-once processing guarantees. Use with exactly-once is not possible.

The properties must be configured for both consumers and producers.

Since:
5.3

  • Constructor Details

  • Method Details

    • listResources

      @Nonnull public Collection<DataConnectionResource> listResources()
      Description copied from interface: DataConnection
      Returns list of DataConnectionResources accessible via this DataConnection.

      It is not strictly required that the data connection lists all resources; a resource can be used even if it is not listed. For example, the list of resources in Oracle database might not include tables available through a database link. In fact, it might list no resources at all, perhaps if the security in the target system prevents reading of such a list.

      The returned list contains up-to-date list of resources. Any changes (added or removed resources) must be reflected in subsequent calls to this method.

    • resourceTypes

      @Nonnull public Collection<String> resourceTypes()
      Description copied from interface: DataConnection
      Returns the list of possible values for DataConnectionResource.type(), that will be returned when DataConnection.listResources() is called. Returned values are case-insensitive, e.g. DataConnectionResource.type() may return MY_RES and this method my_res.

    • newConsumer

      @Nonnull public <K, V> org.apache.kafka.clients.consumer.Consumer<K,V> newConsumer()
      Creates new instance of KafkaConsumer based on the DataConnection configuration. Always creates a new instance of the consumer because KafkaConsumer is not thread-safe. The caller is responsible for closing the consumer instance.
      Returns:
      consumer instance

    • newConsumer

      @Nonnull public <K, V> org.apache.kafka.clients.consumer.Consumer<K,V> newConsumer(Properties properties)
      Creates new instance of KafkaConsumer based on the DataConnection configuration and given properties parameter. Always creates a new instance of the consumer because KafkaConsumer is not thread-safe. The caller is responsible for closing the consumer instance.
      Parameters:
      properties - mapping properties to merge with data connection options.
      Returns:
      consumer instance

    • getProducer

      @Nonnull public <K, V> org.apache.kafka.clients.producer.KafkaProducer<K,V> getProducer(@Nullable String transactionalId)
      Returns an instance of KafkaProducer based on the DataConnection configuration.

      The caller is responsible for closing the producer instance. For non-shared producers the producer will be closed immediately upon calling close. For shared producers the producer will be closed when all users close the returned instance and this DataConnection is released by calling DataConnectionBase.release()

      Parameters:
      transactionalId - transaction id to pass as 'transactional.id' property to a new KafkaProducer instance, must be null for shared producer

    • getProducer

      @Nonnull public <K, V> org.apache.kafka.clients.producer.KafkaProducer<K,V> getProducer(@Nullable String transactionalId, @Nonnull Properties properties)
      Returns an instance of KafkaProducer based on the DataConnection configuration and given properties.

      The caller is responsible for closing the producer instance. For non-shared producers the producer will be closed immediately upon calling close. For shared producers the producer will be closed when all users close the returned instance and this DataConnection is released by calling DataConnectionBase.release()

      Parameters:
      transactionalId - transaction id to pass as 'transactional.id' property to a new KafkaProducer instance, must be null for shared producer
      properties - properties. E.g, SQL mappings provide separate options, and they should be merged with data connection's properties. These properties have higher priority than data connection properties. If KafkaDataConnection is shared, then HazelcastException would be thrown.

    • destroy

      public void destroy()
      Description copied from interface: DataConnection
      Called by the member when shutting down. Should unconditionally close all connections and release resources.