com.hazelcast.jet

Interface JetInstance

public interface JetInstance

Represents either an instance of a Jet server node or a Jet client instance that connects to a remote cluster.

Since:

3.0

Method Summary

Modifier and Type	Method and Description
JetCacheManager	getCacheManager() Obtain the JetCacheManager that provides access to JSR-107 (JCache) caches configured on a Hazelcast Jet cluster.
Cluster	getCluster() Returns information about the cluster this Jet instance is part of.
JetConfig	getConfig() Returns the configuration for this Jet member.
HazelcastInstance	getHazelcastInstance() Returns the underlying Hazelcast IMDG instance used by Jet.
Job	getJob(long jobId) Returns the job with the given id or null if no such job could be found.
default Job	getJob(String name) Returns the active or last submitted job with the given name or null if no such job could be found.
List<Job>	getJobs() Returns all submitted jobs including running and completed ones.
List<Job>	getJobs(String name) Returns all jobs submitted with the given name, ordered in descending order by submission time.
default JobStateSnapshot	getJobStateSnapshot(String name) Returns the JobStateSnapshot object representing an exported snapshot with the given name.
default Collection<JobStateSnapshot>	getJobStateSnapshots() Returns the collection of exported job state snapshots stored in the cluster.
<E> IList<E>	getList(String name) Returns the distributed list instance with the specified name.
<K,V> IMap<K,V>	getMap(String name) Returns the distributed map instance with the specified name.
String	getName() Returns the name of the Jet instance.
<T> Observable<T>	getObservable(String name) Returns an Observable instance with the specified name.
Collection<Observable<?>>	getObservables() Returns a list of all the Observables that are active.
<E> ITopic<E>	getReliableTopic(String name) Returns a distributed reliable topic instance with the specified name.
<K,V> ReplicatedMap<K,V>	getReplicatedMap(String name) Returns the replicated map instance with the specified name.
SqlService	getSql() Returns the Hazelcast SQL service.
default Job	newJob(DAG dag) Creates and returns a Jet job based on the supplied DAG.
Job	newJob(DAG dag, JobConfig config) Creates and returns a Jet job based on the supplied DAG and job configuration.
default Job	newJob(Pipeline pipeline) Creates and returns an executable job based on the supplied pipeline.
Job	newJob(Pipeline pipeline, JobConfig config) Creates and returns a Jet job based on the supplied pipeline and job configuration.
Job	newJobIfAbsent(DAG dag, JobConfig config) Creates and returns a Jet job based on the supplied DAG and job configuration.
Job	newJobIfAbsent(Pipeline pipeline, JobConfig config) Creates and returns a Jet job based on the supplied pipeline and job configuration.
LightJob	newLightJob(DAG dag) Submits a job defined in the Core API.
LightJob	newLightJob(Pipeline p) Submits a light job for execution.
default <T> Observable<T>	newObservable() Returns a new observable with a randomly generated name
void	shutdown() Shuts down the current instance.

Method Detail

getName

@Nonnull
String getName()

Returns the name of the Jet instance.

getHazelcastInstance

@Nonnull
HazelcastInstance getHazelcastInstance()

Returns the underlying Hazelcast IMDG instance used by Jet. It will be either a server node or a client, depending on the type of this JetInstance.

getCluster

@Nonnull
Cluster getCluster()

Returns information about the cluster this Jet instance is part of.

getConfig

@Nonnull
JetConfig getConfig()

Returns the configuration for this Jet member. This method is not available on client instances.

newJob

@Nonnull
default Job newJob(@Nonnull
                            DAG dag)

Creates and returns a Jet job based on the supplied DAG. Jet will asynchronously start executing the job.

newJob

@Nonnull
default Job newJob(@Nonnull
                            Pipeline pipeline)

Creates and returns an executable job based on the supplied pipeline. Jet will asynchronously start executing the job.

newJob

@Nonnull
Job newJob(@Nonnull
                    DAG dag,
                    @Nonnull
                    JobConfig config)

Creates and returns a Jet job based on the supplied DAG and job configuration. Jet will asynchronously start executing the job.

If the name in the JobConfig is non-null, Jet checks if there is an active job with equal name, in which case it throws JobAlreadyExistsException. Job is active if it is running, suspended or waiting to be run; that is it has not completed or failed. Thus there can be at most one active job with a given name at a time and you can re-use the job name after the previous job completed.

See also newJobIfAbsent(com.hazelcast.jet.core.DAG, com.hazelcast.jet.config.JobConfig).

Throws:

JobAlreadyExistsException - if there is an active job with an equal name

newJob

@Nonnull
Job newJob(@Nonnull
                    Pipeline pipeline,
                    @Nonnull
                    JobConfig config)

Creates and returns a Jet job based on the supplied pipeline and job configuration. Jet will asynchronously start executing the job.

If the name in the JobConfig is non-null, Jet checks if there is an active job with equal name, in which case it throws JobAlreadyExistsException. Job is active if it is running, suspended or waiting to be run; that is it has not completed or failed. Thus there can be at most one active job with a given name at a time and you can re-use the job name after the previous job completed.

See also newJobIfAbsent(com.hazelcast.jet.core.DAG, com.hazelcast.jet.config.JobConfig).

Throws:

JobAlreadyExistsException - if there is an active job with an equal name

newJobIfAbsent

@Nonnull
Job newJobIfAbsent(@Nonnull
                            DAG dag,
                            @Nonnull
                            JobConfig config)

Creates and returns a Jet job based on the supplied DAG and job configuration. Jet will asynchronously start executing the job.

If the name in the JobConfig is non-null, Jet checks if there is an active job with equal name. If there is, it will join that job instead of submitting a new one. Job is active if it is running, suspended or waiting to be run; that is it has not completed or failed. In other words, this method ensures that the job with this name is running and is not running multiple times in parallel.

This method is useful for microservices deployment when each package contains a jet member and the job and you want the job to run only once. But if the job is a batch job and runs very quickly, it can happen that it executes multiple times, because the job name can be reused after a previous execution completed.

If the job name is null, a new job is always submitted.

See also newJob(com.hazelcast.jet.core.DAG).

newJobIfAbsent

@Nonnull
Job newJobIfAbsent(@Nonnull
                            Pipeline pipeline,
                            @Nonnull
                            JobConfig config)

Creates and returns a Jet job based on the supplied pipeline and job configuration. Jet will asynchronously start executing the job.

If the name in the JobConfig is non-null, Jet checks if there is an active job with equal name. If there is, it will join that job instead of submitting a new one. Job is active if it is running, suspended or waiting to be run; that is it has not completed or failed. In other words, this method ensures that the job with this name is running and is not running multiple times in parallel.

This method is useful for microservices deployment when each package contains a jet member and the job and you want the job to run only once. But if the job is a batch job and runs very quickly, it can happen that it executes multiple times, because the job name can be reused after a previous execution completed.

If the job name is null, a new job is always submitted.

See also newJob(com.hazelcast.jet.core.DAG).

newLightJob

@Nonnull
LightJob newLightJob(Pipeline p)

Submits a light job for execution. This kind of job is focused on reducing the job startup and teardown time: only a single operation is used to deploy the job instead of 2 for normal jobs. Limitation of light jobs:

• no job configuration, that means no processing guarantee, no custom classes or job resources
• no metrics after job completion
• no visibility in getJobs() or in Management Center (this will be added later)
• failures will be only reported to the caller and logged in the cluster logs, but no trace of the job will remain in the cluster after it's done
• RestartableException doesn't restart the job, but it will fail

It substantially reduces the overhead for jobs that take milliseconds to complete.

A light job will not be cancelled if the client disconnects. It's potential failure will be only logged in member logs.

newLightJob

@Nonnull
LightJob newLightJob(DAG dag)

Submits a job defined in the Core API.

See newLightJob(Pipeline).

getJobs

@Nonnull
List<Job> getJobs()

Returns all submitted jobs including running and completed ones. Currently does not include light jobs.

getJob

@Nullable
Job getJob(long jobId)

Returns the job with the given id or null if no such job could be found. Currently it returns null also for light jobs.

getJobs

@Nonnull
List<Job> getJobs(@Nonnull
                           String name)

Returns all jobs submitted with the given name, ordered in descending order by submission time. The active job is always first. Empty list will be returned if no job with the given name exists. The list includes completed jobs, but currently does not include light jobs.

getJob

@Nullable
default Job getJob(@Nonnull
                             String name)

Returns the active or last submitted job with the given name or null if no such job could be found. The returned job can be already completed.

getJobStateSnapshot

@Nullable
default JobStateSnapshot getJobStateSnapshot(@Nonnull
                                                       String name)

Returns the JobStateSnapshot object representing an exported snapshot with the given name. Returns null if no such snapshot exists.

getJobStateSnapshots

@Nonnull
default Collection<JobStateSnapshot> getJobStateSnapshots()

Returns the collection of exported job state snapshots stored in the cluster.

getSql

@Beta
 @Nonnull
SqlService getSql()

Returns the Hazelcast SQL service.

The service is in beta state. Behavior and API might change in future releases. Binary compatibility is not guaranteed between minor or patch releases.

Hazelcast can execute SQL statements using either the default SQL backend contained in the Hazelcast IMDG code, or using the Jet SQL backend in this package. The algorithm is this: we first try the default backend, if it can't execute a particular statement, we try the Jet backend.

For proper functionality the hazelcast-jet-sql.jar has to be on the class path.

The text below summarizes Hazelcast Jet SQL features. For a summary of the default SQL engine features, see the superclass documentation.

Overview

Hazelcast Jet is able to execute distributed SQL statements over any Jet connector that supports the SQL integration. Currently those are:

• local IMaps (writing only)
• Apache Kafka topics
• Files (local and remote, reading only)

Each connector specifies its own serialization formats and a way of mapping the stored objects to records with column names and SQL types. See the individual connectors for details.

In the first release we support a very limited set of features, essentially only reading and writing from/to the above connectors and projection + filtering. Currently these are unsupported: joins, grouping, aggregation. We plan to support these in the future.

Full Documentation

The full documentation of all SQL features is available at https://jet-start.sh/docs/sql/.

Returns:

SQL service

Since:

4.4

See Also:

SqlService

getMap

@Nonnull
<K,V> IMap<K,V> getMap(@Nonnull
                                String name)

Returns the distributed map instance with the specified name.

It's possible to use the map as a data source or sink in a Jet Pipeline, using Sources.map(String) or Sinks.map(String) and the change stream of the map can be read using Sources.mapJournal(String, JournalInitialPosition).

Parameters:

name - name of the distributed map

Returns:

distributed map instance with the specified name

getReplicatedMap

@Nonnull
<K,V> ReplicatedMap<K,V> getReplicatedMap(@Nonnull
                                                   String name)

Returns the replicated map instance with the specified name.

A replicated map can be used for enriching a stream, see GeneralStage.mapUsingReplicatedMap(String, FunctionEx, BiFunctionEx).

Parameters:

name - name of the distributed map

Returns:

distributed map instance with the specified name

getList

@Nonnull
<E> IList<E> getList(@Nonnull
                              String name)

Returns the distributed list instance with the specified name.

It's possible to use the link as a data source or sink in a Jet Pipeline, using Sources.list(String) or Sinks.list(String).

Parameters:

name - name of the distributed list

Returns:

distributed list instance with the specified name

getReliableTopic

@Nonnull
<E> ITopic<E> getReliableTopic(@Nonnull
                                        String name)

Returns a distributed reliable topic instance with the specified name.

Parameters:

name - name of the distributed topic

Returns:

distributed reliable topic instance with the specified name

Since:

4.0

getCacheManager

@Nonnull
JetCacheManager getCacheManager()

Obtain the JetCacheManager that provides access to JSR-107 (JCache) caches configured on a Hazelcast Jet cluster.

Note that this method does not return a JCache CacheManager

Returns:

the Hazelcast Jet JetCacheManager

See Also:

JetCacheManager

getObservable

@Nonnull
<T> Observable<T> getObservable(@Nonnull
                                         String name)

Returns an Observable instance with the specified name. Represents a flowing sequence of events produced by jobs containing observable sinks.

Multiple calls of this method with the same name return the same instance (unless it was destroyed in the meantime).

In order to observe the events register an Observer on the Observable.

Parameters:

name - name of the observable

Returns:

observable with the specified name

Since:

4.0

newObservable

@Nonnull
default <T> Observable<T> newObservable()

Returns a new observable with a randomly generated name

Since:

4.0

getObservables

@Nonnull
Collection<Observable<?>> getObservables()

Returns a list of all the Observables that are active. By "active" we mean that their backing Ringbuffer has been created, which happens when either their first Observer is registered or when the job publishing their data (via observable sinks) starts executing.

shutdown

void shutdown()

Shuts down the current instance. If this is a client instance, it disconnects the client. If this is a member instance, it gracefully terminates the jobs running on it and, if so configured, restarts them after this instance has shut down. When shutting down the entire cluster, it is a good practice to manually suspend all the jobs so that they don't get restarted multiple times as each member shuts down.

The call blocks until the instance is actually down.

Note: If you call this.getHazelcastInstance().shutdown(), it will cause all the jobs that run on this member to be forcefully terminated, without creating a terminal snapshot. After the cluster stabilizes again, Jet will restart them from the last snapshot that was created some time ago.