Package com.hazelcast.jet.pulsar

Class PulsarSources

java.lang.Object
com.hazelcast.jet.pulsar.PulsarSources
public final class PulsarSources extends Object
Contains builders for creating Pulsar stream sources.
Since:
6.0

  • Method Details

    • pulsarConsumerBuilder

      @Nonnull public static <M, T> PulsarConsumerBuilder<M,T> pulsarConsumerBuilder(@Nonnull List<String> topics, @Nonnull SupplierEx<org.apache.pulsar.client.api.PulsarClient> connectionSupplier, @Nonnull SupplierEx<org.apache.pulsar.client.api.Schema<M>> schemaSupplier, @Nonnull FunctionEx<org.apache.pulsar.client.api.Message<M>,T> projectionFn)
      Returns a builder object that offers a step-by-step fluent API to build a custom Pulsar consumer StreamSource for the Pipeline API.

      Pulsar consumer source is a distributed, timestamped StreamSource which reads messages from Pulsar topics for data ingestion to Jet pipelines. This source does not have fault-tolerance support. It uses the Consumer API of the Pulsar client. It can be used to subscribe partitioned topics. It uses higher level abstraction of Pulsar that is called "shared subscription" that allows multiple consumers to consume from the topics at the same time. The messages are sent round-robin to each connected consumer. Broker determines which consumer will receive a message from which topic partition. It does not require one-to-one mapping between partitions and consumers. Multiple consumers can get messages from same partition. With this source, the message ordering is not preserved.

      Example usage: 

      

 StreamSource<String> pulsarSource = PulsarSources.pulsarConsumerBuilder(
          Arrays.asList(topicName, topicName2 ...),
          () -> PulsarClient.builder()
                            .serviceUrl("pulsar://exampleserviceurl")
                            .build(), // Client Supplier
          () -> Schema.BYTES, // Schema Supplier Function
          x -> new String(x.getData(), StandardCharsets.UTF_8)
                                       // Projection function that converts
                                       // receiving bytes to String
                                       // before emitting.
          ).build();

  Pipeline pipeline = Pipeline.create();
  StreamStage<Status> srcStage = p.readFrom(pulsarSource);
      Type Parameters:
      M - the type of the message read by PulsarConsumer
      T - the type of data emitted from StreamSource
      Parameters:
      topics - the topics to consume, at least one is required
      connectionSupplier - Pulsar client supplier
      schemaSupplier - supplies the schema for consuming messages
      projectionFn - converts a Pulsar message to an emitted item.
      Returns:
      PulsarConsumerBuilder that used to create a StreamSource
      Since:
      6.0

    • pulsarConsumerBuilder

      @Nonnull public static <M, T> PulsarConsumerBuilder<M,T> pulsarConsumerBuilder(@Nonnull String topic, @Nonnull SupplierEx<org.apache.pulsar.client.api.PulsarClient> connectionSupplier, @Nonnull SupplierEx<org.apache.pulsar.client.api.Schema<M>> schemaSupplier, @Nonnull FunctionEx<org.apache.pulsar.client.api.Message<M>,T> projectionFn)
      See the pulsarConsumerBuilder(List, SupplierEx, SupplierEx, FunctionEx) It gets a single String for the topic name in case it should read only from a single topic.
      Type Parameters:
      M - the type of the message read by PulsarConsumer
      T - the type of data emitted from StreamSource
      Parameters:
      topic - the single topic to consume
      connectionSupplier - Pulsar client supplier
      schemaSupplier - supplies the schema for consuming messages
      projectionFn - converts a Pulsar message to an emitted item.
      Returns:
      PulsarConsumerBuilder that used to create a StreamSource
      Since:
      6.0

    • pulsarConsumer

      @Nonnull public static <M, T> StreamSource<T> pulsarConsumer(@Nonnull String topic, @Nonnull SupplierEx<org.apache.pulsar.client.api.PulsarClient> connectionSupplier, @Nonnull SupplierEx<org.apache.pulsar.client.api.Schema<M>> schemaSupplier, @Nonnull FunctionEx<org.apache.pulsar.client.api.Message<M>,T> projectionFn)
      Convenience for pulsarConsumerBuilder(String, SupplierEx, SupplierEx, FunctionEx). It creates a basic Pulsar consumer that connects the topic by using Pulsar client.

      Type Parameters:
      M - the type of the message read by pulsarReader
      T - the type of data emitted from StreamSource
      Parameters:
      topic - the single topic to consume
      connectionSupplier - Pulsar client supplier
      schemaSupplier - supplies the schema for consuming messages
      projectionFn - converts a Pulsar message to an emitted item.
      Returns:
      StreamSource
      Since:
      6.0

    • pulsarReaderBuilder

      @Nonnull public static <M, T> PulsarReaderBuilder<M,T> pulsarReaderBuilder(@Nonnull String topic, @Nonnull SupplierEx<org.apache.pulsar.client.api.PulsarClient> connectionSupplier, @Nonnull SupplierEx<org.apache.pulsar.client.api.Schema<M>> schemaSupplier, @Nonnull FunctionEx<org.apache.pulsar.client.api.Message<M>,T> projectionFn)
      Returns a builder object that offers a step-by-step fluent API to build a custom Pulsar reader StreamSource for the Pipeline API.

      Pulsar reader is a fault-tolerant timestamped StreamSource which reads messages from Pulsar topics for data ingestion to Jet pipelines. It uses the Reader API of the Pulsar client. It cannot be used in the partitioned topics.

      Example usage: 

      

 StreamSource<String> pulsarSource = PulsarSources.pulsarReaderBuilder(
          topicName,
          () -> PulsarClient.builder()
                            .serviceUrl("pulsar://exampleserviceurl")
                            .build(), // Client Supplier
          () -> Schema.BYTES, // Schema Supplier Function
          x -> new String(x.getData(), StandardCharsets.UTF_8)
                                       // Projection function that converts
                                       // receiving bytes to String
                                       // before emitting.
          ).build();

  Pipeline pipeline = Pipeline.create();
  StreamStage<Status> srcStage = p.readFrom(pulsarSource);
      Type Parameters:
      M - the type of the message read by pulsarReader
      T - the type of data emitted from StreamSource
      Parameters:
      topic - the single topic to consume
      connectionSupplier - Pulsar client supplier
      schemaSupplier - supplies the schema for consuming messages
      projectionFn - converts a Pulsar message to an emitted item.
      Returns:
      PulsarReaderBuilder that used to create a StreamSource
      Since:
      6.0

    • pulsarReader

      @Nonnull public static <M, T> StreamSource<T> pulsarReader(@Nonnull String topic, @Nonnull SupplierEx<org.apache.pulsar.client.api.PulsarClient> connectionSupplier, @Nonnull SupplierEx<org.apache.pulsar.client.api.Schema<M>> schemaSupplier, @Nonnull FunctionEx<org.apache.pulsar.client.api.Message<M>,T> projectionFn)
      Convenience for pulsarReaderBuilder(String, SupplierEx, SupplierEx, FunctionEx). It creates a basic Pulsar reader that connects the topic by using Pulsar client.

      Type Parameters:
      M - the type of the message read by pulsarReader
      T - the type of data emitted from StreamSource
      Parameters:
      topic - the single topic to consume
      connectionSupplier - Pulsar client supplier
      schemaSupplier - supplies the schema for consuming messages
      projectionFn - converts a Pulsar message to an emitted item.
      Returns:
      StreamSource
      Since:
      6.0