Documentation¶

Slipstream provides a data-flow model to simplify development of stateful streaming applications.

Simplicity: parallelizing processing, mapping sources to sinks
Freedom: allowing arbitrary code free of limiting abstractions
Speed: optimized and configurable defaults to get started quickly

Consume any AsyncIterable source: Kafka, Streaming API’s, Python generators, Cache updates. Produce to any Callable: Kafka, RocksDB, API’s, Databases. Perform any arbitrary stateful operation using regular Python code: filters, joins, aggregations, pairwise or batching. Detect and handle dependency stream downtimes: pausing, catching up, resuming, and making corrections.

Demo¶

Slipstream components interoperate with basic python building blocks, making it easy to craft framework-like features. For instance, while timers aren’t included, you can whip one up effortlessly (AsyncIterable):

from asyncio import run, sleep

async def timer(interval=1.0):
    while True:
        await sleep(interval)
        yield

We’ll use print as our sink (Callable):

print

Let’s send our mascot “🐟 – blub” downstream:

from slipstream import handle, stream

@handle(timer(), sink=[print])
def handler():
    yield '🐟 - blub'

run(stream())


# 🐟 - blub
# 🐟 - blub
# 🐟 - blub
# ...

Some things that stand out:

We’ve created an Async Iterable source timer() (not generating data, just triggering the handler)
We used slipstream.handle to bind the sources and sinks to the handler function
We yielded 🐟 - blub, which is sent to all the Callable sinks (just print in this case)
Running slipstream.stream starts the flow from sources via handlers into the sinks

The data-flow model that simplifies development of stateful streaming applications!

Contents¶

Proceed by interacting with Kafka and caching application state in: getting started.