Changelog

All notable changes to Streamform are recorded here. The format follows Keep a Changelog and the project uses Semantic Versioning; until 1.0, a minor version may contain breaking changes, and each is marked Breaking in its entry.

Each release section is copied verbatim into the GitHub release notes by the release workflow, so write entries for the people who will run the release, not for the people who wrote it.

Unreleased

Added

• Event time on Flink (DEC-009 decision 6): a source with event_time gets WATERMARK FOR <column> AS <column> - INTERVAL '<delay>' SECOND in its CREATE TABLE and, for JSON, 'json.timestamp-format.standard' = 'ISO-8601'; with late_events: drop the table is created as <source>_raw and the name models read becomes a view that keeps only rows at or after CURRENT_WATERMARK; a model’s retention becomes a STATE_TTL hint on its aggregate, and build and apply warn that on Flink it is enforced in processing time — approximate, and wrong under replay. JSON timestamps on Kafka are read in Flink’s ISO-8601 form, yyyy-MM-ddTHH:mm:ss[.SSS] without a zone. The integration run gains a differential test over an event-time source with in-order events; lateness and expiry are not asserted there.
• Event time runs in the simulator (DEC-009 decisions 3 and 5): a source with event_time keeps a watermark — the largest event time seen minus its watermark delay, never moving backwards, or a fixture’s - watermark: <timestamp> item, whichever is later. An event behind the watermark when it is delivered is late: with late_events: drop it is dropped and counted, with keep it is delivered as if on time. A model’s retention expires a key’s state once the watermark reaches its last event time plus the retention — silently, so the next event for the key starts over as an insert and an upsert sink keeps the stale row until then. streamform test prints N late events dropped, and the fixture’s late: list must name every dropped event, exactly and in delivery order: a dropped event that is not listed fails the fixture with a message saying how to acknowledge it, as does a listed event that was delivered. events processed counts delivered events. test --json gains late_dropped and late rows per fixture (defaulted fields, report_version stays 1).
• Event time is declared and carried: a source in sources.yml may name its event_time column (a timestamp), its watermark (bounded out-of-orderness, a duration such as 5m, default 0s), and late_events (drop or keep, default drop); a model may declare retention (a duration, enforced in event time by the simulator; see the entry above). explain prints time=processing or time=event(<column>) watermark=… late=… on every Source and time attribute=<column> on every node that carries one; inspect-plan prints the same in the Source’s definition and a time field, which is omitted when there is none, so a plan without event time is byte-identical to one written by v0.4.0 and every recorded digest stands. A model’s retention appears in its state fingerprint’s retention= (which has said unbounded since Phase 1) and therefore moves its state digest and only that digest. check rejects an event_time that is not a timestamp column, a watermark or late_events without event_time, and a retention on a model with no event-time source upstream. Fixtures accept RFC 3339 timestamps (2026-08-30T10:00:00Z) for timestamp columns beside the integer form, - watermark: <timestamp> items in given, and a late: list (DEC-009).
• Every release publishes streamform-examples.tar.gz beside the binaries, covered by checksums.txt: the example projects, and the local Kafka and Flink environment as flink-local/. The source repository is private, so this is how the documented first project and the Flink guide are reproduced from public artifacts; the README, the install page, and the guides download it instead of cloning.
• The formats are published: spec/ in the repository holds, for every document the binary reads or writes, its JSON Schema (generated from the binary with mise run spec; CI fails when it drifts), real examples produced from the example projects, and a page of prose; the whole directory is mirrored to glyf-data/streamform-spec on every release. docs/protocol.md states the rules every document and every --json follow: a version field first, newer versions and unknown fields refused, exit codes 0/1/2, stdout holds only the document, values rendered after typing.
• streamform check --json, streamform test --json, and streamform graph --json print one versioned document each (check_version, report_version, graph_version, all 1) and nothing else on stdout; exit codes are unchanged. The test report carries every comparison row the text output shows, with values on both sides rendered after typing against the model’s schema — decimals as strings, so nothing is lost to a float, timestamps as RFC 3339 — and an update’s before-image. check --json prints a document even when the project fails to load (loaded: false, exit 2); graph --json prints the document with its cycles on a reference cycle and exits 1, where the text form refuses.
• streamform schema <document> prints the JSON Schema (draft 2020-12) of a document this build reads or writes — project, sources, fixture, plan-manifest, applied-record, diff, check-report, test-report, graph — generated from the same types the binary uses, so the schema cannot disagree with the binary; streamform schema --list names every document with its version and where it appears. --version N is refused when this build speaks another version. A project file can name its schema for editor completion with # yaml-language-server: $schema=<file>.

Fixed

• Retention counts from the latest event time a key has seen. The simulator counted from whichever event arrived last, so an out-of-order event (inside the watermark delay, or kept by late_events: keep) could bring a key’s expiry forward and drop state that a more recent event had just touched. A write now extends a key’s life or leaves it alone, as it does under Flink’s state TTL.
• A fixture event with no value in the event-time column fails with event N has no `<column>`; source `<source>` runs on event time, so every event needs one, where it used to say plan cannot be simulated. Messages for constructs the simulator does not run say is not supported yet, not in v0.1.
• JSON sinks on Flink write decimals as plain numbers. Flink’s JSON format strips trailing zeros and falls back to scientific notation by default, so a total of 10 reached Kafka as 1E+1, 100 as 1E+2, and 0.000000027 as 2.7E-8. Generated sink tables now set 'json.encode.decimal-as-plain-number' = 'true' (for upsert-kafka, on both the key and the value format), and the same values arrive as 10.000000000, 100.000000000, and 0.000000027, at the column’s scale. Anyone parsing a sink topic as text sees different bytes for the same number; a JSON parser sees the same value. build output changes for every project with a JSON sink.

Changed

• The documentation says what exists: Homebrew, signed artifacts, a container image, and further targets are planned and not available, as are the testing library, the Flink runner image, and the GitHub Action; the only command that opens a network connection is apply, to the SQL Gateway it is given.
• examples/order-pipeline declares event time: orders gains ordered_at: timestamp, event_time: ordered_at, and watermark: 5m; every fixture event carries a timestamp, and tests/late_order.yml shows a late order dropped and acknowledged. docs/event-time.md (new) walks through event time, watermarks, late events, and retention with three runnable fixtures, and what Flink does with each; the site gains a Time concept page and the testing guide a late-events section.
• Breaking (terms): Streamform is a closed-source binary, free to use locally and in production with no account and no limit; LICENSE is now the terms of use rather than the MIT licence (no copy of the MIT-licensed tree was ever distributed). The file formats, the testing library, the Flink runner image, and the GitHub Action are open source under Apache-2.0 in their own repositories. Releases are published to glyf-data/streamform-releases; cargo install --git is no longer an install path.

0.4.0 - 2026-08-27

Streamform now says what an upgrade costs before it is deployed: streamform plan compares the running plan with the desired one and classifies every change (SAFE, COMPATIBLE, STATE MIGRATION REQUIRED, BACKFILL REQUIRED, STATE INCOMPATIBLE), streamform apply records what it applied, and streamform.yml can say that a model was renamed so the diff compares the same node.

Added

• streamform plan [--target NAME] [--against FILE] [--json] compares the running plan with the desired plan and classifies every change as SAFE, COMPATIBLE, STATE MIGRATION REQUIRED, BACKFILL REQUIRED, or STATE INCOMPATIBLE, most severe first, with the fields that differ and a reason per node. A keyed node whose upstream definition changed, even in another model, is raised to BACKFILL REQUIRED, since its state was accumulated under the previous definition. Exit 0 when the application can continue from its existing state, 1 otherwise, 2 when the comparison could not run. Documented in docs/plan-diff.md.
• streamform apply writes the plan it applied, with the time, version, backend, and job ids, to .streamform/applied/<target>.json, and prints the verdict against that record before applying again. It does not refuse on any verdict yet: the Flink backend starts a new job with empty state on every apply, and refusing waits for job lifecycle.
• renamed_from: <model> and nodes: { <kind>.<ordinal>: { was: <id> } } on a model in streamform.yml tell plan which running node a desired node was, so a renamed model or a shifted ordinal compares as the same node. streamform check rejects malformed or contradictory hints and a hint for a node the plan does not have; plan warns about a hint that names no running node.

0.3.0 - 2026-08-26

Streamform is the project’s name from this release on, and every stateful node now has an identity that survives an upgrade: streamform inspect-plan prints stable node ids, SHA-256 state and schema digests, and changelog contracts, streamform build writes them to build/plan.json, and a documentation site lives under site/. The release artifacts are the first named streamform-<target>.tar.gz.

Added

• streamform inspect-plan [MODEL] [--json] prints the stable identity of the plan: every node’s id, definition, output schema with its digest, changelog contract, and, for a keyed operator, its key, accumulators, retention, and state digest. With --json it prints the plan manifest, a versioned document (manifest_version: 1) holding the same for the whole application plus an application digest. Documented in docs/plan-identity.md.
• streamform build writes the plan manifest to build/plan.json next to the generated application, so every build carries the identity it was built from.
• A documentation site under site/ (mise run docs, mise run docs-serve): getting started, concepts, guides, and reference, with the founding documents included from docs/. A landing page lives at site/landing/index.html. The Docs workflow builds both on every change; the site is run locally, nothing deploys it.

Changed

• Breaking: the project is renamed to Streamform. The binary is streamform, the crates are streamform-*, the project manifest file is streamform.yml, and the release artifacts are streamform-<target>.tar.gz. Releases v0.1.0 and v0.2.0 shipped under the previous name; their entries below are written with the current names.
• Breaking: an aggregate column without an alias is now named by Streamform (sum(amount), count(*)) rather than by the SQL planner (sum(orders.amount)), and an aggregate column keeps its accumulator type through a projection (decimal(38,2) for a SUM over decimal(10,2), where the planner said decimal(20,2)). Generated Flink SQL changes accordingly for unaliased aggregates; fixtures that name such a column must use the new name. Aliased columns are unaffected.
• streamform explain prints a digest= line under a keyed operator’s fingerprint: the first 12 characters of the SHA-256 of the fingerprint text.
• Breaking: streamform explain --json prints the model’s slice of the plan manifest (manifest_version, model, nodes) instead of the previous hand-written shape; every field of the plan is now present, and fingerprint_digest is state.digest.

Fixed

• State fingerprint digests were computed with the standard library’s hasher, whose output may change between Rust releases. They are now SHA-256 over the fingerprint’s canonical text, so a toolchain upgrade cannot mark existing state incompatible.

0.2.0 - 2026-08-25

Streamform understands an application, not a statement, and runs it on Flink. ref() chains models into a dependency graph; streamform graph draws it; streamform build --backend flink turns the plan into Flink SQL and streamform apply submits it through the SQL Gateway; mise run integration proves on real Kafka and Flink 2.0.2 that the simulator and Flink agree. Coverage fixtures now run every aggregate, operator, and column type through SQL.

Added

• ref('<model>') in model SQL reads another model of the same project. Models form a dependency graph; streamform check reports an unknown ref(), a reference cycle, and a name used by both a source and a model. streamform explain <model> shows the upstream model as from=<model> on the first operator, and streamform explain with no model prints every model in dependency order.
• A fixture may target a model that reads other models: given events go to the source at the top of the chain and expect is asserted on the target model’s output, after every model in between.
• A model’s Sink shows its declared materialization and, when a sink: block is configured, the connector and target.
• streamform build --backend flink [--target NAME] [--stdout] generates the application as Flink SQL: a CREATE TABLE per source, a CREATE TEMPORARY VIEW per model, a CREATE TABLE sink_<model> per declared sink (upsert-kafka with a primary key for upsert models), and one EXECUTE STATEMENT SET. No Java or Python is generated. Documented in docs/flink-backend.md.
• streamform apply [--target NAME] builds the application and submits it through the target’s Flink SQL Gateway REST API, statement by statement, printing the job id; a rejected statement stops the apply with Flink’s reason and exit code 1.
• integration/docker-compose.yml runs Kafka and Flink 2.0.2 with the Kafka connector locally; mise run integration applies the examples to it and checks that the sink topics carry exactly what the simulator’s changelog implies, tombstones included.
• targets: in streamform.yml names where an application is built for and applied to: backend, flink.gateway, kafka.bootstrap_servers, kafka.startup. streamform check reports a Flink target missing its sections.
• streamform graph prints the model dependency graph as an indented tree; streamform graph --format dot emits Graphviz.
• examples/order-pipeline: the three-model example, orders → clean_orders → customer_metrics → big_customers, whose last fixture asserts an insert, an update, and a delete from one filter over an upsert model.

Changed

• Breaking for callers of the Rust API: streamform_sql::compile and compile_all are replaced by compile_application, which returns one plan for the whole project; streamform_sim::run takes the model to run. The CLI is unchanged.
• GROUP BY over a model that emits updates is refused at planning (“aggregation over an updating stream”), since aggregating updates needs retractions that are not supported yet.
• Unsupported constructs are reported as “not supported yet” rather than “not supported in v0.1”.
• An upsert model’s declared key must match the key its plan produces: streamform check now reports a key that disagrees with the GROUP BY (previously accepted silently), and a global aggregate is declared with no key (previously impossible).

Fixed

• COUNT(*) failed to plan (“the expression *”) in every release so far; nothing had exercised it through SQL. A coverage fixture project now runs every aggregate, operator, and column type through SQL.

0.1.0 - 2026-08-24

The first release: deterministic testing and inspection for streaming SQL semantics. No Flink, Kafka, dbt, Python, windows, watermarks, joins, ref(), or deployment yet.

Added

• streamform check loads and validates a project — streamform.yml, sources.yml, models/*.sql, tests/*.yml — compiles every model, and reports every problem in one run with did you mean suggestions.
• streamform explain <model> prints the model’s streaming plan: operators, boundedness, changelog mode, state requirement, key, and a state fingerprint naming each accumulator and its type. --json for machine consumption.
• streamform test [--model NAME] runs every fixture through a deterministic in-process simulator and asserts the exact changelog, record by record. Prints PASS/FAIL with event and record counts, a side-by-side diff on failure, and exits 1 when any fixture fails.
• The Streaming IR (streamform-ir): source, filter, projection, keyed aggregation, and sink operators; append and upsert changelog modes; stable node identity <model>::<kind>::<ordinal>; state fingerprints rendered through Streamform-owned type names so a dependency upgrade cannot move them.
• The SQL frontend (streamform-sql): DataFusion plans a model’s SELECT, and lowering produces the IR. Supported: column references, literals, comparison and arithmetic operators, AND/OR/NOT, IS [NOT] NULL, CAST, WHERE, GROUP BY over columns, HAVING, and the aggregates sum, count, min, max. Anything else fails naming the construct. ORDER BY over an unbounded source is rejected as meaningless.
• The simulator (streamform-sim): single-threaded, in-memory, no wall clock; operators defined over changelogs so a filter after an aggregate emits inserts and deletes at the predicate boundary; exact fixed-point decimals; checked arithmetic that reports overflow rather than wrapping.
• Project format: source('<name>') in SQL; per-model materialized: append | upsert with key; source column types int32, int64, float64, decimal, decimal(precision,scale), string, boolean, timestamp; fixtures with given events and expect records carrying op: insert | update | delete. Documented in docs/project-format.md.
• Release builds for macOS (Intel and Apple silicon) and Linux x86_64 with checksums.txt, and cargo install --git from the tag.