Streamform — Vision

What is Streamform?

Streamform is a Rust-native framework for building, testing, planning, and deploying continuously running data applications from declarative SQL models.

The user describes:

source
+
transformation
+
materialization

Streamform turns that into:

continuous dataflow

An execution backend such as Apache Flink runs that dataflow in production.

SQL models
   +
source definitions
   +
materialization
        │
        ▼
      Streamform
        │
        ▼
Streaming IR
    /       \
Simulator   Flink

Streamform itself is not another streaming database and initially is not another distributed streaming engine.

Its value sits above execution engines:

Give data engineers and software engineers a software-engineering workflow for long-running stateful data applications.

Problem

There is a large gap between writing a transformation and operating it continuously.

A batch data engineer can comfortably write:

SELECT
    customer_id,
    SUM(amount) AS total_spend
FROM orders
GROUP BY customer_id

But turning that logic into:

Kafka
   ↓
continuous processing
   ↓
persistent state
   ↓
safe updates
   ↓
production deployment

requires understanding:

• Kafka
• Flink
• state management
• changelogs
• checkpoints
• watermarks
• event time
• serialization
• deployment
• savepoints
• upgrades
• recovery

The application also behaves differently from batch SQL.

A batch query:

starts
↓
processes data
↓
finishes

A streaming application:

starts
↓
processes events
↓
accumulates state
↓
continues running
↓
changes over time
↓
must survive deployments
↓
may run for years

That lifecycle is the real problem Streamform should solve.

User Experience

A project might eventually look like:

fraud-pipeline/
├── streamform.yml
├── sources.yml
├── models/
│   ├── clean_transactions.sql
│   └── account_activity.sql
└── tests/
    └── account_activity.yml

A source:

sources:
  commerce.transactions:
    connector: kafka
    topic: transactions
    format: json

A model:

SELECT
    account_id,
    COUNT(*) AS transaction_count,
    SUM(amount) AS total_amount
FROM source('commerce.transactions')
GROUP BY account_id

A materialization:

models:
  account_activity:
    materialized: upsert

    key:
      - account_id

    sink:
      connector: kafka
      topic: account-activity

The developer workflow:

streamform check
streamform test
streamform explain
streamform plan
streamform apply
streamform inspect

Core Insight

SQL alone is not a streaming application.

A streaming application consists of:

INPUT SEMANTICS

What is the input?
Bounded file?
Kafka stream?
CDC changelog?

        +

COMPUTATION

What transformation should happen?

        +

MATERIALIZATION SEMANTICS

How should continuously changing results be represented?
Append?
Upsert?
Retract?
Queryable state?

Streamform owns the semantics connecting these three pieces.

First Principle: Streaming Is About Change

Consider:

SELECT
    customer_id,
    SUM(amount)
FROM orders
GROUP BY customer_id

Input events might be append-only:

customer=42 amount=10

customer=42 amount=18

But the output relation changes:

INSERT
customer=42 total=10

UPDATE
customer=42 total=28

Streamform must understand this distinction.

Therefore concepts such as:

Append
Upsert
Update
Retract

are first-class concepts.

They are not implementation details of Flink.

Second Principle: State Is Part of the Program

The aggregation:

GROUP BY customer_id

requires the runtime to remember previous values.

Conceptually:

state

customer 42
    count = 18
    total = 902

customer 91
    count = 4
    total = 88

Therefore Streamform must understand:

which operators require state

how that state is keyed

what schema the state has

whether state can survive a new program version

State must be represented explicitly in Streamform’s semantic model.

Third Principle: Streaming Programs Must Be Testable

A user should not need:

Kafka
+
Flink cluster
+
Docker
+
sleep(10)
+
grep logs

to test transformation logic.

Streamform should provide deterministic streaming tests.

Example:

model: customer_metrics

given:
  - customer_id: 42
    amount: 10

  - customer_id: 42
    amount: 18

expect:
  - op: insert
    customer_id: 42
    total_spend: 10

  - op: update
    customer_id: 42
    total_spend: 28

Then:

streamform test

executes completely in-process.

fixture events
      ↓
Streaming IR
      ↓
Rust simulator
      ↓
exact changelog

No external infrastructure.

Fourth Principle: Running Applications Have a Lifecycle

Streamform’s long-term differentiation should not be merely:

compile SQL to Flink.

The difficult production question is:

What happens when the SQL changes while the application already has hundreds of gigabytes of state?

Example:

Previous:

GROUP BY customer_id

New:

GROUP BY customer_id, country

Streamform should eventually tell the developer:

STATE INCOMPATIBLE

Model:
customer_metrics

Grouping key changed:

- customer_id
+ customer_id, country

Existing keyed state cannot be safely reused.

Required action:
REBUILD

Another change could result in:

SAFE

Existing state can be reused.

Or:

BACKFILL REQUIRED

This requires Streamform to understand the lifecycle of the program, not merely its syntax.

Fifth Principle: Execution Engines Are Backends

Initial architecture:

                    Streamform Streaming IR
                         /       \
                        /         \
             Simulator             Flink

Later:

                    Streamform Streaming IR
                  /         |          \
                 /          |           \
          Simulator       Flink       Streamform Runtime

Flink provides mature production execution.

Streamform owns:

• authoring semantics
• streaming semantics
• state semantics
• changelogs
• validation
• testing
• lifecycle
• plan compatibility

This prevents Streamform from becoming coupled to Flink.

Why Rust?

The implementation should be Rust-first.

Rust is particularly appropriate because the project can eventually involve:

• SQL compilation
• Arrow-native memory
• query-plan transformations
• event processing
• state stores
• async networking
• Kafka
• storage
• concurrency
• checkpointing
• distributed execution

It also lets Streamform ship as a single native CLI without requiring a Python runtime.

Initial installation should aim toward:

brew install streamform

or:

cargo install streamform

Python bindings or:

pip install streamform

can be introduced later if there is a concrete Python integration need.

Python should not be required for the initial architecture.

What Streamform Is Not

Streamform is initially not:

• a streaming database
• a Kafka replacement
• a Flink replacement
• a warehouse
• a generic workflow orchestrator
• a dbt fork
• an agent framework
• a hosted runtime (a hosted control plane that never runs user workloads is not excluded; the founding text read “a hosted SaaS platform”)
• an API framework
• a distributed Rust streaming engine

Those boundaries are deliberate.

Initial User

The initial user is an engineer comfortable with:

SQL
data models
Kafka concepts
Git
CI

but who does not want streaming application development to require deep Flink expertise.

Later users may include:

• analytics engineers
• data platform teams
• backend engineers
• real-time product teams

Longer-Term Direction

The project can evolve naturally:

Phase 1
streaming compiler + simulator

        ↓

Phase 2
production Flink execution

        ↓

Phase 3
safe application upgrades

        ↓

Phase 4
runtime contracts

        ↓

Phase 5
native Rust runtime

        ↓

Phase 6
queryable materialized state

        ↓

Phase 7
real-time API serving

Eventually:

events
  ↓
Streamform model
  ↓
continuous state
  ↓
REST / gRPC / Arrow Flight

could allow a data engineer to turn continuously maintained data into production application state without separately building an API service.

That is a future extension, not the initial objective.

Product Thesis

Streamform’s thesis is:

Continuously running data applications should be developed with the same confidence, testability, versioning, and deployment discipline as normal software.

The product is therefore not primarily:

easier streaming SQL.

It is:

a software development system for stateful streaming applications.