slipstream.checkpointing¶
Slipstream checkpointing.
Attributes¶
Classes¶
Per-dependency pulse result. |
|
Track the dependent stream state to recover from downtime. |
|
Track one stream against the streams it depends on. |
Functions¶
|
Heartbeat dependencies and pulse the dependent around a handler. |
Module Contents¶
- class slipstream.checkpointing.Downtime[source]¶
Bases:
dictPer-dependency pulse result.
Falsy when empty. A single entry compares equal to that value, so
downtime == timedelta(...)still holds for one leader.
- class slipstream.checkpointing.Dependency(name: str, dependency: collections.abc.AsyncIterable[Any], downtime_threshold: Any = timedelta(minutes=10), downtime_check: slipstream.utils.AsyncCallable[[Checkpoint, Dependency], Any] | None = None, recovery_check: slipstream.utils.AsyncCallable[[Checkpoint, Dependency], bool] | None = None, marker: collections.abc.Callable[..., Any] | str | None = None, state: collections.abc.Callable[[Any, dict[str, Any]], dict[str, Any]] | None = None)[source]¶
Track the dependent stream state to recover from downtime.
The dependency name should not be changed once created, it is used to persist the dependency in the cache.
>>> async def emoji(): ... for emoji in '🏆📞🐟👌': ... yield emoji >>> Dependency('emoji', emoji()) {'checkpoint_state': None, 'checkpoint_marker': None}
- save(cache: slipstream.interfaces.ICache, cache_key_prefix: str, checkpoint_state: Any, checkpoint_marker: datetime.datetime) None[source]¶
Save checkpoint state to cache.
- load(cache: slipstream.interfaces.ICache, cache_key_prefix: str) None[source]¶
Load checkpoint state from cache.
- class slipstream.checkpointing.Checkpoint(dependent: collections.abc.AsyncIterable[Any], *leaders: collections.abc.AsyncIterable[Any] | Dependency, dependencies: collections.abc.AsyncIterable[Any] | Dependency | list[collections.abc.AsyncIterable[Any] | Dependency] | None = None, name: str | None = None, on_downtime: collections.abc.Callable[[Checkpoint, Dependency], Any] | None = None, on_recovery: collections.abc.Callable[[Checkpoint, Dependency], Any] | None = None, cache: slipstream.interfaces.ICache | None = None, cache_key_prefix: str = '_', pause_dependent: bool = True, downtime_threshold: datetime.timedelta | None = None, marker: collections.abc.Callable[..., Any] | str | None = None, state: collections.abc.Callable[[Any, dict[str, Any]], dict[str, Any]] | None = None)[source]¶
Track one stream against the streams it depends on.
>>> async def emoji(): ... for emoji in '🏆📞🐟👌': ... yield emoji >>> dependent, dependency = emoji(), emoji() >>> checkpoint = Checkpoint( ... dependent, ... Dependency('dependency', dependency), ... name='dependent', ... )
Pass a marker and bind the checkpoint to call
heartbeatandcheck_pulseautomatically:>>> checkpoint = Checkpoint( ... dependent, ... Dependency('dependency', dependency, marker='timestamp'), ... name='dependent', ... marker='timestamp', ... ) >>> from slipstream import handle >>> @handle(checkpoint) ... async def dependent_handler(msg, checkpoint=None): ... yield msg
If no cache is provided, the checkpoint lasts only for this process.
- async heartbeat(marker: datetime.datetime | Any, dependency_name: str | None = None, checkpoint_state: Any = None) dict[source]¶
Update checkpoint to latest state.
- Args:
- marker (datetime | Any): Typically the event timestamp that is
compared to the event timestamp of a dependent stream.
- dependency_name (str, optional): Required when there are multiple
dependencies to specify which one the heartbeat is for.
checkpoint_state: Complete caller-selected dependency state.
- async check_pulse(marker: datetime.datetime | Any, checkpoint_state: Any = None, **kwargs: Any) Any | None[source]¶
Update state that can be used as checkpoint.
- Args:
- marker (datetime | Any): Typically the event timestamp that is
compared to the event timestamp of a dependency stream.
checkpoint_state: Complete caller-selected dependent state. kwargs (Any): Any information that can be used for reprocessing any
incorrect data that was sent out during downtime of a dependency stream, stored in state.
- Returns:
None when every dependency is healthy. Otherwise a
Downtimemap of name → check (orTrueif still down but not over threshold). One leader still compares equal to its timedelta /True.
- slipstream.checkpointing.bind_checkpoint(f: collections.abc.Callable[..., Any], handler: collections.abc.Callable[..., collections.abc.Awaitable[Any]], checkpoint: Checkpoint) collections.abc.Callable[..., collections.abc.Awaitable[Any]][source]¶
Heartbeat dependencies and pulse the dependent around a handler.