Testing with fixtures

A fixture delivers events to a model and asserts the exact changelog the model emits. streamform test runs every fixture under tests/ through the deterministic simulator, in-process, with no services.

Anatomy of a fixture

model: big_customers

given:
  - customer_id: 42
    amount: 60
  - customer_id: 42
    amount: 70
  - customer_id: 42
    amount: 10
  - customer_id: 42
    amount: -50

expect:
  - op: insert
    customer_id: 42
    total_spend: 130
  - op: update
    customer_id: 42
    total_spend: 140
  - op: delete
    customer_id: 42
    total_spend: 140

model names the model under test.

given lists input events, delivered in order to the source at the top of the model’s chain. big_customers reads ref('customer_metrics'), which reads ref('clean_orders'), which reads source('orders'); the events are orders rows, and the changelog asserted is big_customers’ own, after every model in between has run. Columns the source declares but an event omits are null; a column the source does not declare is an error.

expect is the changelog, record by record, in order. The comparison is exact: the same number of records, and record i must have the op and every output column value of expect[i]. An update is asserted by its after-image. Each record names exactly the model’s output columns, and streamform check reports a misspelled or missing column before anything runs.

Values are typed against the schema: an integer 28 matches a decimal column holding 28.000000000; 0.1234 against decimal(38,2) is an error, not a rounding; null matches only null.

What the fixture above proves

Customer 42’s total climbs to 130 (they enter the filter: insert), then 140 (update), then a refund takes it to 90 and they leave (delete). That third record is the reason changelogs are first-class: a WHERE over an updating input must emit a delete when a row stops matching, and a consumer of the big-customers topic depends on receiving it.

Running

streamform test                     # every fixture
streamform test --model big_customers

Each fixture prints its model, PASS or FAIL, the events processed, and the records emitted. A failure adds a side-by-side listing with the first difference named:

customer_metrics

FAIL
  --> tests/customer_metrics.yml

2 events processed
2 changelog records emitted

  #  expected                              actual
  1  insert customer_id=42 total_spend=10  insert customer_id=42 total_spend=10
  2  update customer_id=42 total_spend=27  update customer_id=42 total_spend=28
     ^ total_spend: expected 27, got 28

failed: 1 of 1 fixture

Exit codes: 0 when every fixture passes, 1 when any fails, 2 when the project cannot be tested at all (it does not load, check would report errors, or --model names a model that does not exist).

Determinism

The simulator is single-threaded and has no wall clock. The same fixture and the same plan produce the same changelog on every run; the repository’s own test suite runs the acceptance fixture a hundred times and asserts identical output. That is what makes a fixture a specification rather than a flaky observation, and what lets the simulator serve as the oracle the Flink backend is compared against.

Practical notes

A project with no tests/ directory passes honestly: ok: no fixtures under tests/.
Fixtures are checked before they run. streamform check compiles every model and validates every expectation’s shape against the model’s output columns, so a typo fails fast with the column list in the message.
The - event: {…} form for an item in given reads well beside - watermark: … items, which advance an event-time source’s watermark explicitly. See below.

Late events and time

On a source that declares event_time, a fixture also controls the clock. Timestamps are RFC 3339 instants (2026-08-30T10:00:00Z); - watermark: <timestamp> items advance the watermark between events; and an event that arrives behind the watermark is late — dropped and counted under the default late_events: drop, and then it must be listed under late:, exactly and in order, or the fixture fails:

model: customer_metrics

given:
  - event: { customer_id: 42, amount: 10, ordered_at: "2026-08-30T10:00:00Z" }
  - event: { customer_id: 42, amount: 20, ordered_at: "2026-08-30T10:10:00Z" }
  - event: { customer_id: 42, amount: 5,  ordered_at: "2026-08-30T10:01:00Z" }   # behind the 10:05 watermark

expect:
  - { op: insert, customer_id: 42, total_spend: 10 }
  - { op: update, customer_id: 42, total_spend: 30 }

late:
  - { customer_id: 42, amount: 5, ordered_at: "2026-08-30T10:01:00Z" }

2 events processed
2 changelog records emitted
1 late event dropped

Leave the late: section out and the failure names the event and the two ways to resolve it — acknowledge it, or set late_events: keep on the source. A model with retention expires a key’s state once the watermark passes its last event plus the retention, silently; a fixture shows it by the key’s next event arriving as an insert rather than an update. The three canonical fixtures — a late event, the watermark advancing, state expiring — are walked through in Event time.

Keyboard shortcuts