The streaming plan
Everything Streamform does goes through one representation: the streaming plan. check validates against it, explain prints it, test executes it in the simulator, build generates Flink SQL from it, and inspect-plan describes its identity. Because every command reads the same plan, what you test is what you deploy.
From SQL to a plan
A model’s SQL is parsed and planned by DataFusion, then immediately lowered into Streamform’s own intermediate representation and DataFusion is not consulted again. The IR is deliberately small: five operators and a seven-variant expression tree covering exactly what Streamform supports today. SQL that does not fit fails at check, naming the construct, instead of reaching a backend that would run it differently from the simulator.
| Operator | What it does | Holds state |
|---|---|---|
| Source | Reads a declared source | no |
| Filter | Keeps rows matching a predicate (WHERE, HAVING) | no |
| Project | Computes output columns | no |
| Aggregate | Groups rows and maintains one accumulator set per key (GROUP BY) | yes |
| Sink | Ends every model: its materialized relation, with an optional external destination | no |
What every node knows
Each node carries three properties that are first-class in Streamform rather than details of an engine:
- Boundedness: whether the node’s output ever ends. Derived from the source’s connector and carried through. See Boundedness.
- Changelog mode: how the output changes over time:
Append,Upsert(key), orRetract. See Changelogs. - State requirement:
Stateless, orKeyedwith a fingerprint of exactly what is stored. See State.
explain prints them under each operator:
Aggregate
key=customer_id
aggregates=sum(amount)
boundedness=Unbounded
changelog=Upsert(customer_id)
state=Keyed
fingerprint=aggregate key=[customer_id] accumulators=[sum(amount): decimal(38,9)] retention=unbounded
digest=4aead023aa01
One plan per application
A project with several models compiles to one plan. Models appear in dependency order, each as a chain ending in its Sink, and ref('m') is an edge from a model’s first operator to m’s Sink. A changelog mode crosses that edge: a model reading an upsert model receives updates, not fresh facts, and its own operators are planned accordingly. See Applications of many models.
Nothing engine-specific inside
The plan contains nothing that belongs to Flink, and the Flink backend is generated from it alone. That is a design rule with a test behind it: if a feature ever needs an engine concept inside the plan, the plan is what gets revisited, not the backend. It is also what makes the simulator an honest oracle for Flink, and what will let a native runtime consume the same plan later.