public final class WatermarkPolicies extends Object
Modifier and Type | Method and Description |
---|---|
static DistributedSupplier<WatermarkPolicy> |
limitingLag(long lag)
Maintains a watermark that lags behind the top observed timestamp by the
given amount.
|
static DistributedSupplier<WatermarkPolicy> |
limitingLagAndDelay(long lag,
long maxDelayMs)
Maintains a watermark that lags behind the top observed timestamp by at
most the given amount and is additionally guaranteed to reach the
timestamp of any given event within
maxDelayMs of system time
after observing it. |
static DistributedSupplier<WatermarkPolicy> |
limitingLagAndLull(long lag,
long maxLullMs)
Maintains watermark that lags behind the top timestamp by the amount
specified with
lag . |
static DistributedSupplier<WatermarkPolicy> |
limitingTimestampAndWallClockLag(long timestampLag,
long wallClockLag)
Maintains watermark that lags behind the top timestamp by at most
timestampLag and behind wall-clock time by at most wallClockLag . |
@Nonnull public static DistributedSupplier<WatermarkPolicy> limitingLag(long lag)
lag
- the desired difference between the top observed timestamp
and the watermark@Nonnull public static DistributedSupplier<WatermarkPolicy> limitingLagAndDelay(long lag, long maxDelayMs)
maxDelayMs
of system time
after observing it.
Keep in mind that the scope of the "top observed timestamp" is only the
items received by an individual processor: the watermark of a processor
using this policy will never advance beyond the timestamp it observed in
isolation from the rest of the system. In the case of severe stream skew
a given processor's watermark may lag significantly behind that of other
processors and cause delays when getting coalesced with them in
downstream processors. To overcome this you can also configure JobConfig.setMaxWatermarkRetainMillis
.
lag
- upper bound on the difference between the top observed timestamp and the
watermarkmaxDelayMs
- upper bound (in milliseconds) on how long it can take for the
watermark to reach any observed event's timestamp@Nonnull public static DistributedSupplier<WatermarkPolicy> limitingTimestampAndWallClockLag(long timestampLag, long wallClockLag)
timestampLag
and behind wall-clock time by at most wallClockLag
. It assumes that the event timestamp is in milliseconds
since Unix epoch and will use that fact to correlate it with wall-clock
time acquired from the underlying OS. Note that wall-clock time is
non-monotonic and sudden jumps that may occur in it will cause temporary
disruptions in the functioning of this policy.
In most cases the limitingLagAndLull
policy should be preferred; this is a backup option
for cases where some substreams may never see an event.
timestampLag
- maximum difference between the top observed timestamp
and the watermarkwallClockLag
- maximum difference between the current value of
System.currentTimeMillis
and the watermark@Nonnull public static DistributedSupplier<WatermarkPolicy> limitingLagAndLull(long lag, long maxLullMs)
lag
. Assumes that the event timestamp is given
in milliseconds and will use that fact to correlate it with the passage
of system time. There is no requirement on any specific point of origin
for the timestamp, i.e., the zero value can denote any point in time as
long as it is fixed.
When the defined maxLullMs
period elapses without observing more
events, watermark will start advancing in lockstep with system time
acquired from the underlying OS's monotonic clock.
If no event is ever observed, watermark will advance from the initial
value of Long.MIN_VALUE
. Therefore this policy can be used only
when there is a guarantee that each substream will emit at least one
event that will initialize the timestamp. Otherwise the empty substream
will hold back the processing of all other substreams by keeping the
watermark below any realistic value. Configuring WatermarkGenerationParams.idleTimeoutMillis()
will cope with this.
lag
- the desired difference between the top observed timestamp
and the watermarkmaxLullMs
- maximum duration of a lull period before starting to
advance watermark with system timeCopyright © 2018 Hazelcast, Inc.. All rights reserved.