Project layout

A project is a directory. Every file in it is a contract Streamform keeps: the formats are versioned, and a change to any of them is a documented change.

streamform.yml     project name, format version, per-model materialization, targets
sources.yml        the external inputs and their schemas
models/*.sql       one SQL model per file; the file stem is the model name
tests/*.yml        fixtures, one per file
build/             output of `streamform build`; not source, keep it out of version control

streamform.yml

Names the project, pins the format version (1), and configures models and targets. A model exists because models/<name>.sql exists; the entry under models: only configures it. A model with no entry is append with no key.

name: order-pipeline
version: 1

models:
  customer_metrics:
    materialized: upsert
    key: [customer_id]
  big_customers:
    materialized: upsert
    key: [customer_id]
    sink:
      connector: kafka
      topic: big-customers
      format: json

targets:
  local:
    backend: flink
    flink:
      gateway: http://localhost:8083
    kafka:
      bootstrap_servers: kafka:9092

• materialized is append (the default) or upsert, and must agree with what the SQL produces: a GROUP BY produces an updating result and requires upsert with a key equal to the grouping columns.
• sink adds an external destination for the model’s changelog. Every model ends in a Sink in its plan whether or not it declares one; the Sink is what fixtures assert on and what ref() in another model reads. A sink: block is what makes it leave the application.
• targets name the places an application is built for and applied to. See Running on Apache Flink.

sources.yml

Each source has a connector, a location, a format, and a schema. The connector decides whether the input ends: kafka is unbounded, file is bounded. That is derived, never guessed from the SQL.

sources:
  orders:
    connector: kafka
    topic: orders
    format: json
    schema:
      customer_id: int64
      amount: decimal(12,2)

Types: int32, int64, float64, decimal (which is decimal(38,9)), decimal(precision,scale), string, boolean, timestamp. The declared scale of a decimal is what state keeps: a SUM over decimal(10,2) accumulates decimal(38,2).

models/*.sql

One SELECT per file. Read a source with source('<name>') and another model with ref('<model>'); a plain table name is not valid, because the dependency graph would not know about it.

The supported surface today: column references, literals, comparison and arithmetic, AND/OR/NOT, IS [NOT] NULL, CAST, WHERE, GROUP BY over columns, HAVING, and sum, count, min, max. Anything else fails at check naming the construct, rather than reaching a backend that would run it differently.

tests/*.yml

A fixture names a model, gives it events, and states the exact changelog it must emit. Testing with fixtures walks through it; the project format reference is exact about every field and every comparison rule.