Skip to main content

Model Pipelines

A folder of .wv files can work like a dbt project. You define models, mark the ones to persist, and wvlet build builds them in dependency order. Wvlet works out the dependency graph from model references, so you don't need ref() calls or a separate DAG file.

Materializing Models​

Add a with { } config to a model to choose how wvlet build persists it:

-- models/staging.wv
model stg_orders with { materialized: 'view' } = {
from raw.orders
where status != 'cancelled'
}

-- models/marts.wv
model customer_totals with { materialized: 'table', schema: 'marts' } = {
from stg_orders
group by customer_id
agg amount.sum as total
}
materialized:wvlet build writesReferencing the model
ephemeral (default)NothingInlines the model's query, as plain models always do
viewcreate or replace view (like save as view)Reads the view during a build
tablecreate or replace table ... as (like save to)Reads the table during a build

Other config keys:

KeyDescription
schemaTarget schema. Defaults to the profile's current schema
aliasTarget relation name. Defaults to the model name
engineProfile connector to build the model on. Defaults to the default engine
retries, timeout, retry_delay, backoff, max_retry_delayRetry and timeout settings for the model's build step, with the same meaning as stage configuration

Configs are checked at compile time. An unknown key or materialization is an error. So is materializing a model with parameters, because such a model has no single result to persist.

Static types, inlined outside a build​

A model reference keeps the model's static type wherever it appears. Wvlet never looks the persisted relation up in the live database to type-check it.

  • During wvlet build, a model that reads a materialized model reads its persisted table or view, as dbt's ref() does. This also applies when the read goes through ephemeral models without parameters.
  • Everywhere else (wvlet run, the REPL, spec tests), models are inlined as before. Queries therefore never fail on a table that has not been built yet, and never read stale results.

Building Models​

# Build every materialized model in dependency order
wvlet build

# Print the build order and generated SQL without running anything
wvlet build --dry-run

# Build one model, its downstream models, or its upstream models
wvlet build -s customer_totals
wvlet build -s stg_orders+
wvlet build -s +customer_totals

# Several selectors, comma-separated
wvlet build -s stg_orders,customer_totals
SelectorSelects
mModel m only. The materialized models it reads must already be built
m+m and every model downstream of it
+mm and every model upstream of it
+m+Both directions

A model builds only after every selected model it reads. The build order lists models by dependency level, in definition order within a level. A selector that names no model (for example -s "") is an error.

Materialized models must not form a dependency cycle. That includes a model that reads itself, directly or through ephemeral models. wvlet build reports such a cycle as a path (for example Cyclic model dependency: a -> b -> a) and builds nothing, even when the selection avoids the cycle.

wvlet build uses the same options as wvlet flow: -w <folder>, --profile, --catalog, --schema, and --run-store.

A build runs on the flow executor as a flow named build, with one stage per model:

  • Models that don't depend on each other build in parallel.
  • A failed model is retried as its config says. Models downstream of a failure are skipped, and the command exits with a non-zero code.
  • Runs are recorded, so wvlet flow session list and wvlet flow session show <run id> show past builds.
  • If a selection reads a materialized model that has never been built, the build stops before running anything and names the models to build first.

To recover from a failure, fix the model and run wvlet build again, or rerun only the failed part with a selector such as wvlet build -s failed_model+.

Feature Parity with dbt​

dbtWvlet
Models and ref()model definitions referenced by name, type-checked at compile time
Materializations: view, table, ephemeralwith { materialized: ... }
Incremental modelsKeyed upserts with append to t on keys. An incremental materialization is planned
dbt build --selectwvlet build -s
Sourcestable declarations bound to catalog.schema. Freshness checks are planned
Seedsfrom 'file.csv' with save to t if not exists
Snapshots (SCD2)Planned
Singular data teststest assertions
Generic tests (unique, not_null, relationships)Column constraints on table declarations. Running them as data tests is planned
Jinja macrosType-checked building blocks: reusable query fragments (def is_adult = where age >= 18) and scalar functions (def double(x: int): int = x * 2), parameterized models, for loops, interpolated table names (s`name_${x}` ), and import
adapter.dispatchEngine-specific standard library and trait ... in <dialect>
var(), env_var()Flow parameters. Model variables are planned
Pre- and post-hooksexecute statements in flows. Model hooks are planned
Docs and lineage--- doc comments. Generated docs and lineage are planned
Targets and profilesProfiles

Wvlet doesn't add a text-template layer like Jinja. Templates would lose compile-time type checking, and with it the ability to compile one model to several SQL engines.