Streamform — Product roadmap

Status assessed on 2026-09-16 against the local checkout at 6e9e818. This is a source and documentation assessment, not a fresh certification of tests, releases, or external repositories.

This roadmap replaces the original phase-by-phase build order. The vision remains the product thesis; architecture sets the compiler boundaries; the implementation strategy (docs/implementation-strategy.md in the source repository) defines delivery work and acceptance gates. Future behavior described here is planned, not an existing CLI or file-format contract.

Product direction

Streamform should make a stateful streaming application understandable, testable, and safe to change. The initial user is an engineer who knows SQL, Kafka, Git, and CI, but should not need to become a Flink specialist to answer: “What will this application emit, and what happens to its state when I deploy this change?”

The near-term product is a Rust compiler, deterministic simulator, and deployment workflow over Flink. Its differentiator is the connection between tested semantics, explainable change impact, and an observed deployment. Native execution, serving, and a hosted control plane remain options after this workflow earns adoption.

The next product proof is a customer-metrics application that can be tested with late events and windows, deployed once, inspected, and restarted from verified state without duplicate jobs or unexplained loss. A changed grouping key must produce an actionable refusal or explicit rebuild plan, not an implied safe upgrade.

Where the project actually stands

The workspace version remains 0.4.0. Event time is already built; the immediate work is validating and packaging it, not implementing it again. The earlier roadmap’s Phase 0–6 work is largely present, Phase 7 is implemented but unreleased, and Phase 8 begins with tumbling windows.

Delivery order

Milestones are ordered by dependencies and evidence, not dates. v0.5 is the existing release target; later version numbers should be assigned when their gates are met.

Execute M0 first, then M1. Start the bounded M2/M3 feasibility investigations during M1 if capacity permits; they need not delay the window contract, and they must finish before upgrade promises are made. After M1, deliver M2 then M3 before adding sliding/session windows or starting a native runtime. This is work sequencing, not a requirement for parallel agents.

M0 — Release the capabilities already built

Close the v0.5 release around event time and the published protocol. Run the repository checks and real-cluster suite, preserve the documented processing-time retention limitation, and publish reproducible examples with the binary. A public quick start must work without cloning this private source repository.

Audit the release-facing prose: the README and introduction still say event time is absent; the Unreleased changelog contains an earlier intermediate claim that watermark fixtures are refused; contribution/release instructions and ecosystem availability statements are inconsistent with newer distribution choices. Classify external components as verified available, planned, or unavailable. Do not make the release depend on implementing every ecosystem promise; implement or explicitly defer each one.

Exit: a new user on a supported platform installs the release, gets the documented check/explain/test output from downloadable examples, validates the JSON reports against shipped schemas, and can follow the Flink guide with accurate caveats. Release tags and publication remain explicit release actions.

M1 — One complete window capability

Deliver event-time tumbling aggregation with final emission, append input, and fixed positive millisecond durations. Start with a single event-time lineage and existing aggregates. Keep window boundaries, emission, cleanup, and late-data semantics in the generic IR. The window proposal (docs/windows.md in the source repository) supersedes the previous draft and identifies the decisions the feasibility work must settle.

Do not ship a simulator-only interpretation while presenting it as portable to Flink. Reject unsupported combinations before submission. Add a canonical window example, versioned schema changes where needed, stable fingerprints, meaningful plan-diff reasons, and a differential integration test.

Exit: an engineer can predict when a window becomes final, observe no premature output, test boundary and late events without Docker, and run the supported equivalent on Flink. Memory is reclaimed when windows close; fixture exhaustion is not an implicit infinite watermark.

M2 — Make deployment observable and repeatable

Separate desired intent, last submitted deployment, and observed cluster state. A local applied record is evidence of a prior submission, not proof that a job is alive or that its state is restorable.

Add application/target ownership, read-only inspection, an immutable artifact identity, deployment attempts with recoverable state, and a single-writer guard. Repeating the same apply against the same verified running deployment should be a no-op. A timeout after submission is an unknown outcome requiring reconciliation, not permission to start another job. Replacements and rebuilds need explicit modes and sink consequences.

Exit: repeated apply, process interruption, missing jobs, stale records, and ambiguous HTTP outcomes all have tested outcomes. The CLI never announces a healthy deployment solely because a statement was accepted. Define remote gateway TLS, authentication, credential redaction, and supported deployment environments before claiming remote production readiness.

M3 — Turn plan advice into a supported upgrade workflow

First prove stop/savepoint/restore of the identical deployed artifact on the pinned Flink version, including source offsets and externally visible sink behavior. Then enable only the change classes for which the backend can prove physical state compatibility.

Streamform node ids and SHA-256 semantic fingerprints are necessary, but do not identify Flink’s physical operators or serializers. Investigate compiled plans and backend operator identity before selecting the deployment representation. If the current SQL Gateway approach cannot preserve identity, record that constraint and keep unsupported upgrade execution blocked; do not silently rebuild or create a new runtime to avoid the decision.

Keep semantic verdict and executable deployment action separate. A COMPATIBLE semantic change may still have no supported restore path; a STATE MIGRATION REQUIRED result does not mean a migrator exists. Rebuilds require a replayable source range and an explicit sink replacement/cutover policy.

Exit: automated tests demonstrate one supported state-preserving operation, unsupported operations fail before mutation, and partial failures leave sufficient evidence for operator recovery. Do not promise universal rollback or exactly-once delivery from a savepoint alone.

M4 — Continuous contracts

Start with a small set such as not-null and simple predicates, and one deterministic failure policy per release. Define contract placement relative to filters, windows, aggregation, and sink writes. Failure reports should identify the model, contract, and count without unexpectedly exporting event payloads.

Quarantine/dead-letter output is a separate sink with its own schema and delivery semantics. Add it only after its failure and retry behavior is specified. A runtime fail action needs a defined failed-job and restart policy.

Exit: fixture failures and runtime failures are observable and explainable, with a backend capability check for every supported action.

M5 — Expand from observed workloads

Prioritize requests by concrete applications and support cost. The candidate order is retraction-aware aggregation and explicit upsert/CDC inputs; multiple-source fixtures and watermark coordination; bounded interval or temporal joins; then additional windows. This order is provisional and may change with user evidence.

Aggregation over updates needs correct before/after and delete handling, including non-invertible aggregates. Multiple inputs need source-qualified fixtures, watermark minimum/idle-source rules, and bounded state policies before joins. Sliding windows multiply retained state; session windows can merge keys and require new emission and compatibility rules.

Small SQL additions such as CASE, casts, and selected scalar functions may be delivered between milestones when backed by real queries and parity tests. Avoid expanding syntax without executable semantics. dbt import and Python clients belong here only when a real integration need justifies them.

M6 — Keep expensive options conditional

The existing free binary and open-format product direction is retained. External library, runner, Action, and Cloud plans are adoption/distribution work, not substitutes for semantic or lifecycle correctness. Their current availability was not verified in this review.

Measures of progress

Track time from installation to first passing fixture; whether users can explain their changelog and plan verdict; time to diagnose a failed deployment; and successful recoveries without duplicate jobs or lost state. Begin with three representative design-partner applications as a proposed validation cohort, not an adoption claim.

Engineering gates are concrete: unchanged manifests keep their digests where promised; supported backend cases have differential tests; repeated apply creates no additional owned job; restore tests continue accumulated totals rather than restarting them; unsupported behavior fails early and names the limitation.

Window state is bounded in event time, not absolutely bounded in memory: stalled watermarks and unbounded key cardinality still require visibility and limits. Operational resource measurements belong in acceptance evidence, not an unqualified “bounded state” claim.

Mapping from the founding roadmap

For issue preparation, use the stable work-package ids and dependency rules in the implementation strategy (docs/implementation-strategy.md). Create detailed issues for M0–M3 first; keep later work as gated epics until the preceding evidence exists.