Streamform — Architecture
This document describes the founding architecture, with some examples from the initial implementation. For the assessed current baseline, delivery sequence, and upcoming window/lifecycle decisions, see the roadmap; the delivery work behind it is kept in the source repository (docs/implementation-strategy.md). Future-looking examples here do not imply that a command or guarantee is implemented.
Architectural Overview
Streamform Project
│
┌───────────┴───────────┐
│ │
SQL models metadata
sources
sinks
tests
│ │
└───────────┬───────────┘
▼
┌────────────────┐
│ Frontend │
│ │
│ DataFusion │
│ LogicalPlan │
└───────┬────────┘
▼
┌────────────────┐
│ Semantics │
│ │
│ boundedness │
│ state │
│ changelog │
│ keys │
│ time │
└───────┬────────┘
▼
┌────────────────┐
│ Streaming IR │
└───────┬────────┘
│
┌──────────┼───────────┐
▼ ▼ ▼
Simulator Flink Native Rust
future
1. Rust Workspace
Start with a small workspace.
streamform/
├── Cargo.toml
├── README.md
├── AGENTS.md
│
├── docs/
│ ├── vision.md
│ ├── architecture.md
│ └── roadmap.md
│
├── crates/
│ ├── streamform-cli/
│ ├── streamform-project/
│ ├── streamform-sql/
│ ├── streamform-ir/
│ ├── streamform-semantics/
│ └── streamform-sim/
│
└── examples/
└── customer-metrics/
Do not create a crate for every future concept.
Add boundaries only when real code requires them.
2. Project Frontend
Streamform V0 should use its own minimal project format.
Do not make dbt a runtime dependency.
Example:
streamform.yml
sources.yml
models/*.sql
tests/*.yml
Later:
dbt manifest
↓
DbtFrontend
↓
same internal representation
becomes another frontend.
This keeps:
Streamform semantics
independent from:
dbt semantics
3. DataFusion Boundary
Use DataFusion for generic relational machinery.
SQL
↓
DataFusion parser/planner
↓
DataFusion LogicalPlan
↓
Streamform lowering
↓
Streaming IR
DataFusion provides machinery such as:
- SQL parsing
- column resolution
- expression trees
- data types
- coercion
- aggregates
- joins
- logical plans
- Arrow schemas
Streamform should not make DataFusion’s plan canonical.
Lower relatively early.
4. Streaming IR
The Streaming IR is the central architecture.
A possible starting point:
#![allow(unused)]
fn main() {
pub struct StreamingPlan {
pub nodes: Vec<StreamNode>,
}
}#![allow(unused)]
fn main() {
pub struct StreamNode {
pub id: NodeId,
pub operator: Operator,
pub schema: Schema,
pub boundedness: Boundedness,
pub changelog: ChangelogMode,
pub state: StateRequirement,
}
}Initial operators:
#![allow(unused)]
fn main() {
pub enum Operator {
Source(Source),
Filter(Filter),
Project(Project),
Aggregate(Aggregate),
Sink(Sink),
}
}Keep V0 intentionally tiny.
5. Stable Node Identity
Every stateful node should have stable identity.
For example:
#![allow(unused)]
fn main() {
pub struct NodeId(String);
}A node identity should not depend purely on:
position in DAG
because the eventual lifecycle system must compare:
Plan V1
vs
Plan V2
Stable identity enables:
state compatibility
plan diff
migration planning
deployment reconciliation
Design this early.
6. Boundedness
Streamform should explicitly model:
#![allow(unused)]
fn main() {
enum Boundedness {
Bounded,
Unbounded,
}
}Examples:
Parquet snapshot
→ Bounded
Kafka topic
→ Unbounded
This changes what SQL means.
For instance:
ORDER BY event_time
over an infinite Kafka stream cannot produce a globally final ordering.
The semantic layer should catch such cases.
7. Changelog Model
The result of a streaming operator is not always append-only.
Start with something like:
#![allow(unused)]
fn main() {
enum Change {
Insert(Record),
Update {
before: Record,
after: Record,
},
Delete(Record),
}
}The logical capability could separately be:
#![allow(unused)]
fn main() {
enum ChangelogMode {
Append,
Upsert(Key),
Retract,
}
}This distinction becomes foundational for:
- testing
- sinks
- materialized state
- API serving
- subscriptions
8. State Requirements
Represent state explicitly.
#![allow(unused)]
fn main() {
enum StateRequirement {
Stateless,
Keyed {
keys: Vec<Column>,
fingerprint: StateFingerprint,
},
}
}For:
GROUP BY customer_id
derive:
state:
keyed
key:
customer_id
A first simulator implementation can use:
#![allow(unused)]
fn main() {
HashMap<Key, AggregateState>
}There is no need for RocksDB or distributed state initially.
9. State Fingerprint
Introduce the concept early, even if compatibility logic comes later.
Conceptually:
StateFingerprint
operator:
aggregate
key:
customer_id
accumulators:
count: int64
total_spend: decimal
retention:
unbounded
Two program versions can eventually compare these fingerprints.
This powers:
streamform plan
10. Deterministic Simulator
The simulator is the first execution backend.
Architecture:
Fixture Events
│
▼
Source
│
▼
Filter
│
▼
Aggregate
│
▼
Changelog
Properties:
- single process
- initially single-threaded
- in-memory
- deterministic
- no Kafka
- no Flink
- no Docker
- no wall-clock dependency
The simulator is simultaneously:
- a testing engine
- the second consumer of the IR
- the seed of the future Rust runtime
11. Test Model
Example:
model: customer_metrics
given:
- customer_id: 42
amount: 10
- customer_id: 42
amount: 18
expect:
- op: insert
customer_id: 42
total_spend: 10
- op: update
customer_id: 42
total_spend: 28
Then:
streamform test
produces:
customer_metrics
PASS
2 events processed
2 changelog records emitted
The fixture format — how given values are typed against the source schema, what an expect record must contain, and how a failure is rendered — is specified in project-format.md. The comparison is exact: same number of records, same op, same column values, in order.
Tests should eventually support explicitly controlled:
- event order
- event timestamps
- watermarks
- late events
12. Flink Backend
Only after the IR + simulator work should Flink be introduced.
Streaming IR
│
▼
Flink Backend
│
▼
Flink SQL / executable representation
Flink owns:
- distributed execution
- checkpointing
- network shuffle
- recovery
- state backend
- backpressure
- parallelism
Streamform owns:
- semantic interpretation
- validation
- plan structure
- changelog meaning
- state compatibility
- developer workflow
13. Multi-Model DAG
Streamform eventually supports:
source orders
↓
clean_orders
↓
customer_metrics
↓
sink
Streamform’s own authoring syntax can initially provide:
source(...)
ref(...)
These do not need to depend on dbt. As built: a project is one StreamingPlan; every node knows its model; ref('m') is an edge from a model’s first operator to m’s Sink node, so a model is planned once no matter how many read it, and changelog mode crosses the boundary (see docs/project-format.md).
Later the dbt frontend can map dbt’s:
source()
ref()
into the same model.
14. Lifecycle / Reconciler
This should become one of the project’s most important systems.
Commands:
streamform plan
streamform apply
streamform inspect
streamform drain
rather than thinking primarily in terms of:
run
deploy
because a streaming application is persistent.
Conceptually:
Desired Plan
│
▼
compare
▲
│
Running Plan
Output:
SAFE
STATE MIGRATION
STATE INCOMPATIBLE
BACKFILL REQUIRED
15. Native Rust Runtime
The simulator evolves gradually into a runtime.
Progression:
deterministic simulator
↓
Arrow batches
↓
async sources
↓
bounded channels
↓
backpressure
↓
persistent keyed state
↓
checkpointing
↓
recovery
↓
parallel operators
↓
distributed workers
Do not begin with distributed execution.
16. Queryable State
When persistent keyed state is implemented, design it so it can eventually be accessed externally.
Conceptually:
Runtime
│
├── update state
│
└── query state
│
▼
customer_id=42
This makes future serving much easier.
17. API Serving
Later:
Kafka
↓
Streamform computation
↓
queryable keyed state
↓
HTTP / gRPC / Arrow Flight
The serving layer should be an extension of maintained state, rather than a separate duplicated pipeline.
18. Python
Python should be optional and delayed.
Potential future use:
pip install streamform
could provide:
from streamform import Project
project = Project.load(".")
project.test()
Possible implementation:
Rust core
↓
PyO3
↓
Python package
But only add this when:
- notebook integration matters
- Python programmatic APIs matter
- dbt ecosystem integration requires it
The CLI and compiler should remain Rust-native.