Streamform

Streamform is a workflow for building, testing, running, and upgrading stateful streaming applications from declarative SQL models. You write SQL, declare where the input comes from and how the result should be materialized, and Streamform compiles that into a continuous dataflow it can test in-process and run on Apache Flink.

It is one Rust binary, streamform, and a project is a directory of plain files. There is no JVM on your machine, no service to run, and nothing generated that you cannot read.

The one idea everything follows from

Take the most ordinary aggregation there is:

SELECT customer_id, SUM(amount) AS total_spend
FROM source('orders')
GROUP BY customer_id

Over a stream, its input is append-only: an order for customer 42 of 10, then another of 18. But its output is not a growing list. It is a relation that changes:

insert   customer_id=42  total_spend=10
update   customer_id=42  total_spend=28

A SQL statement says nothing about that. It does not say whether orders ends, that the result is keyed by customer_id, that computing it needs the runtime to remember every customer’s total, or what should happen to that memory when you change the query and deploy again. Today that knowledge lives in the person who knows Flink.

Streamform makes it part of the program. A model is the SQL plus three declarations the SQL cannot make:

From those, Streamform derives the rest: whether each step is bounded, what changelog it emits, what state it holds, and a stable identity for that state so a later version can be compared against it.

The workflow

streamform check          validate the project against its compiled plan
streamform explain        read a model as a streaming plan
streamform test           run fixtures through the simulator, assert the changelog
streamform build          generate the application as Flink SQL
streamform apply          submit it through the Flink SQL Gateway
streamform inspect-plan   see the identity and digests an upgrade compares
streamform plan           compare the running plan with the desired one and classify every change

Every command compiles the project to the same plan and works from it, so what you test is what you deploy and what you inspect.

Where to start

• Install the binary and run the first project in about a minute.
• Read the concepts if you want to know what explain is telling you.
• The reference is exact about every command, file, and exit code.
• Why Streamform holds the founding documents, for the reasoning behind the design.

Status

Streamform is early and says so. Version 0.4 ships single- and multi-model applications with append and upsert changelogs, the deterministic simulator, the Flink backend, a stable plan identity for every node, and streamform plan, which classifies every change before it is deployed. Built since, and shipping as version 0.5: event time, with watermarks, late events that a fixture must acknowledge, and retention. Not yet: windows, joins, job lifecycle, dbt and Python frontends. The roadmap is ordered by dependency and evidence, not by date, and each capability lands whole (project format, simulator, Flink, plan diff, documentation) before it is called shipped. The project format is versioned and unknown fields are rejected, so an older build never silently misreads a newer project.