T
- event typepublic final class EventTimePolicy<T> extends Object implements Serializable
timestampFn
: extracts the timestamp from an event in the stream
newWmPolicyFn
: a factory of watermark policy objects.
Refer to its documentation for explanation.
idleTimeoutMillis
: a measure to mitigate the issue with temporary
lulls in a distributed event stream. It pertains to each partition
of a data source independently. If Jet doesn't receive any events
from a given partition for this long, it will mark it as "idle" and let
the watermark in downstream vertices advance as if the partition didn't
exist.
wrapFn
: a function that transforms a given event and its
timestamp into the item to emit from the processor. For example, the
Pipeline API uses this to wrap items into JetEvent
s as a way
to propagate the event timestamps through the pipeline regardless of
the transformation the user does on the event objects themselves.
EventTimeMapper
when implementing a
source processor.Modifier and Type | Field and Description |
---|---|
static long |
DEFAULT_IDLE_TIMEOUT
The default idle timeout in milliseconds.
|
Modifier and Type | Method and Description |
---|---|
static <T> EventTimePolicy<T> |
eventTimePolicy(ToLongFunctionEx<? super T> timestampFn,
ObjLongBiFunction<? super T,?> wrapFn,
SupplierEx<? extends WatermarkPolicy> newWmPolicyFn,
long watermarkThrottlingFrameSize,
long watermarkThrottlingFrameOffset,
long idleTimeoutMillis)
Creates and returns a new event time policy.
|
static <T> EventTimePolicy<T> |
eventTimePolicy(ToLongFunctionEx<? super T> timestampFn,
SupplierEx<? extends WatermarkPolicy> newWmPolicyFn,
long watermarkThrottlingFrameSize,
long watermarkThrottlingFrameOffset,
long idleTimeoutMillis)
Creates and returns a new event time policy.
|
long |
idleTimeoutMillis()
Returns the amount of time allowed to pass without receiving any events
from a partition before marking it as "idle".
|
SupplierEx<? extends WatermarkPolicy> |
newWmPolicyFn()
Returns the factory of the watermark policy objects.
|
static <T> EventTimePolicy<T> |
noEventTime()
Returns an event time policy that results in no timestamping.
|
ToLongFunctionEx<? super T> |
timestampFn()
Returns the function that extracts the timestamp from the event.
|
long |
watermarkThrottlingFrameOffset()
|
long |
watermarkThrottlingFrameSize()
This value together with
watermarkThrottlingFrameOffset()
specify the frame size the watermarks are throttled to. |
ObjLongBiFunction<? super T,?> |
wrapFn()
Returns the function that transforms the received item and its timestamp
into the emitted item.
|
public static final long DEFAULT_IDLE_TIMEOUT
public static <T> EventTimePolicy<T> eventTimePolicy(@Nullable ToLongFunctionEx<? super T> timestampFn, @Nonnull ObjLongBiFunction<? super T,?> wrapFn, @Nonnull SupplierEx<? extends WatermarkPolicy> newWmPolicyFn, long watermarkThrottlingFrameSize, long watermarkThrottlingFrameOffset, long idleTimeoutMillis)
noEventTime()
.timestampFn
- function that extracts the timestamp from the event;
if null, Jet will use the source's native timestampwrapFn
- function that transforms the received item and its
timestamp into the emitted itemnewWmPolicyFn
- factory of the watermark policy objectswatermarkThrottlingFrameSize
- the frame length to which we
throttle watermarks, see watermarkThrottlingFrameSize()
watermarkThrottlingFrameOffset
- the frame offset to which we
throttle watermarks, see watermarkThrottlingFrameOffset()
idleTimeoutMillis
- the timeout after which a partition will be
marked as idle. Use 0 to disable the feature.public static <T> EventTimePolicy<T> eventTimePolicy(@Nullable ToLongFunctionEx<? super T> timestampFn, @Nonnull SupplierEx<? extends WatermarkPolicy> newWmPolicyFn, long watermarkThrottlingFrameSize, long watermarkThrottlingFrameOffset, long idleTimeoutMillis)
noEventTime()
.timestampFn
- function that extracts the timestamp from the event;
if null, Jet will use the source's native timestampnewWmPolicyFn
- factory of the watermark policy objectswatermarkThrottlingFrameSize
- the frame length to which we
throttle watermarks, see watermarkThrottlingFrameSize()
watermarkThrottlingFrameOffset
- the frame offset to which we
throttle watermarks, see watermarkThrottlingFrameOffset()
idleTimeoutMillis
- the timeout after which a partition will be
marked as idle.public static <T> EventTimePolicy<T> noEventTime()
@Nullable public ToLongFunctionEx<? super T> timestampFn()
@Nonnull public ObjLongBiFunction<? super T,?> wrapFn()
@Nonnull public SupplierEx<? extends WatermarkPolicy> newWmPolicyFn()
public long watermarkThrottlingFrameSize()
watermarkThrottlingFrameOffset()
specify the frame size the watermarks are throttled to. Generally it
should match the window slide step used downstream. If there are
multiple sliding windows downstream, use the greatest common denominator
of them.
If this parameter is equal to 0, all watermarks will be suppressed.
Technically, a watermark should be emitted after every increase in event time. Because watermarks are broadcast from each processor to all downstream processors, this will bring some overhead. But the watermarks are only needed for window aggregation and only when a window should close, that is at the frame boundary of a sliding window. To reduce the amount of watermarks on the stream, you can configure to emit only those watermarks that would trigger an emission of a new window.
public long watermarkThrottlingFrameOffset()
public long idleTimeoutMillis()
WatermarkCoalescer.IDLE_MESSAGE
to its
output edges. This signals Jet that the watermark can advance as
if the partition didn't exist.
If you supply a zero or negative value, partitions will never be marked as idle.
Copyright © 2022 Hazelcast, Inc.. All rights reserved.