Skip to main content

Connector - Wvlet Engine

The wvlet engine is an in-process columnar query engine that executes compiled Wvlet plans directly — no SQL generation and no external database. It runs on every platform the CLI supports (JVM, Node.js, and the native binary), reading Parquet, Iceberg, JSON, and CSV data through pure-Scala readers, so it also works where DuckDB is unavailable.

Preview

The engine covers queries over files and registered tables (filters, projections, joins including ASOF joins, aggregations, window functions, subqueries, pivot, unnest, lambdas, show commands, and TPC-H), plus save to, append to, and delete on in-memory tables and save to / append to Parquet, CSV, and JSON files and Iceberg tables, and create table, drop table, truncate, and create / drop schema. Features it doesn't support yet report an error: alter table, views, update, execute, flows, table functions such as read_csv(...), and some engine-specific functions. Map values run but can return results that differ from DuckDB.

Usage​

Select the engine with -t wvlet:

# Query a local file
wvlet run -t wvlet "from 'data/orders.parquet' where o_totalprice > 1000 limit 10"

# Run a query file
wvlet run -t wvlet -f report.wv

Relative file paths resolve against the working folder (-w).

-t wvlet without --host runs the in-process engine; with --host it connects to a remote wvlet server instead.

Registering tables​

Define a profile with a wvlet connector to reference tables by name. Its properties register tables:

  • dataDir: a folder whose data files (.parquet, .json, .jsonl, .csv, .tsv, and their .gz variants) and sub-folders (Iceberg tables or folders of Parquet files) become tables named after the file name without its extension
  • tables: an object mapping table names to paths
~/.wvlet/profiles.json
{
"profiles": [
{
"name": "local-engine",
"connectors": [
{
"name": "wvlet",
"type": "wvlet",
"properties": {
// customer.parquet, orders.parquet, ... become tables customer, orders, ...
"dataDir": "data",
"tables": {
"events": "logs/events.jsonl",
"sales": "warehouse/sales"
}
}
}
]
}
]
}
wvlet run -p local-engine "from orders join customer on o_custkey = c_custkey agg _.count"

Relative paths resolve against the working folder. The compiler sees the schemas of Parquet and Iceberg tables (read from file footers), so their column names are checked as for other engines. JSON and CSV tables are read only when a query scans them.

Saving results​

save to <table> keeps a query result as a table, and append to and delete change it. Without a warehouse, tables live in memory for the rest of the session:

from 'data/orders.parquet'
where o_orderstatus = 'F'
save to finished_orders

-- Add rows; `append to` a table that doesn't exist creates it
from 'data/orders_today.parquet'
where o_orderstatus = 'F'
append to finished_orders

-- Insert-or-update by key: rows with a matching o_orderkey update the existing rows
append to finished_orders on o_orderkey {
from 'data/order_fixes.parquet'
}

-- Remove rows
from finished_orders
where o_totalprice < 100
delete

Appended rows map to the table's columns by position, or to the columns listed in append to t (c1, c2) (the others get their declared default, or NULL), and are cast to the column types. A keyed append to t on k matches columns by name, as SQL MERGE does: matched rows get the new values of the query's columns, and rows with a NULL key always insert. Tables registered from files (dataDir, tables) are read-only: writing to them reports an error instead of changing the files.

Creating tables​

create table makes an empty table from a table declaration, including column defaults and not null, primary key, and unique constraints, which every write to the table checks:

table accounts = {
id: int primary key
email: string unique not null
status: string = 'active'
}

create table accounts
-- Fills status with its default
append to accounts (id, email) {
from [[1, 'alice@example.com']] as t(id, email)
}

truncate accounts
drop table accounts
  • create table t if not exists keeps an existing table, and create or replace table t starts over; a plain create table of an existing table is an error
  • append to a declared table that doesn't exist yet creates it from the declaration first
  • create schema s and save to s.t group tables under a schema; drop schema s with cascade: true drops them, and drop schema s of a schema that still holds tables is an error
  • drop table of a table registered from files only unregisters it; the files stay as they are

Changing tables​

reshape (or SQL alter table) changes a table's columns, keeping its rows:

reshape accounts {
add signup_date: date -- NULL in existing rows
rename status as state
cast id as long -- change a column's type
exclude email -- drop a column
}

rename table accounts to members

SQL alter table also sets or drops a column's default and not null. Tables registered from files can't be changed.

Persistent tables​

Give the connector a warehouse folder to keep tables across sessions, as a DuckDB database file does. Every table written by name (create table, save to t, append to t) goes there and is read back by the next session:

~/.wvlet/profiles.json
{
"name": "wvlet",
"type": "wvlet",
"properties": {
"warehouse": "warehouse"
}
}
  • Each table is an Iceberg table at <warehouse>/<schema>/<table>: tables of the default schema main are under warehouse/main/, and create schema s makes the warehouse/s/ folder. DuckDB, Spark, Trino, and pyiceberg read the tables too
  • Every write is one atomic Iceberg commit: save to t and truncate t replace the rows in a new snapshot (save to may change the columns), append to t adds a snapshot, and delete and append to t on k rewrite only the data files holding the rows they change (copy-on-write)
  • reshape / alter table change the table's metadata only when Iceberg allows it (adding a column without a default, dropping or renaming a column, widening int to long or a decimal's precision, defaults, dropping not null); other changes rewrite the table in one snapshot. Renaming a warehouse table isn't supported yet
  • Declared defaults and not null / primary key / unique constraints are kept in the table's properties and checked by later sessions too
  • drop table deletes the table's folder, and drop schema s with cascade: true the schema's
  • A name that is both a warehouse table and a dataDir / tables entry is an error when the session starts. Tables registered from files stay read-only
  • Without warehouse, tables are in memory only (like DuckDB's :memory:). The browser playground always keeps tables in memory

Writing files​

save to '<path>' writes a query result to a file, and append to '<path>' adds rows to it (creating the file when it doesn't exist). The extension selects the format:

ExtensionFormat
.parquetParquet
.csv, .tsvCSV / TSV with a header row
.json, .jsonl, .ndjsonNewline-delimited JSON
from 'data/orders.parquet'
where o_orderstatus = 'F'
save to 'out/finished_orders.parquet' with compression: 'zstd'

from 'data/orders_today.parquet'
where o_orderstatus = 'F'
append to 'out/finished_orders.parquet'

Options for save to ... with:

  • format: parquet, csv, tsv, or json, overriding the extension
  • compression (Parquet): snappy, zstd, gzip, brotli, lz4, or uncompressed
  • header (CSV, default true) and delimiter (CSV)

Appended rows map to the file's columns by position. CSV, TSV, and JSON lines files get the new rows added at the end, leaving the existing text as is; Parquet files are rewritten with the new rows cast to the file's column types, through a temporary file so a failed append keeps the original. Files are written to local paths only; writing to s3:// or other remote paths isn't supported yet.

Iceberg tables​

The engine reads and writes Apache Iceberg tables in local folders, on every platform, with no Spark or DuckDB. A folder is an Iceberg table when it has a metadata/ folder holding .metadata.json files.

-- Create a table (snapshot 1), partitioned by country
from 'data/events.parquet'
save to 'warehouse/events' with format: 'iceberg', partition_by: 'country'

-- Add a snapshot
from 'data/events_today.parquet'
append to 'warehouse/events'

-- Read the current snapshot, or the one a metadata file records
from 'warehouse/events'
from 'warehouse/events/metadata/v1.metadata.json'
  • save to '<dir>' with format: 'iceberg' creates a table in a new or empty folder. Options: partition_by (comma-separated identity partition columns) and compression. save to on an existing Iceberg table is an error: the engine doesn't replace Iceberg tables yet, so use append to or write to a new folder
  • append to '<dir>' commits the rows as a new snapshot, mapped to the table's columns by position and cast to their types. It works on tables committed as metadata/vN.metadata.json (as the wvlet engine and Hadoop-catalog writers create them); a table that a catalog owns, such as one written by pyiceberg or a REST catalog, is read-only. append to a folder that isn't an Iceberg table is an error, since append to can't name the format
  • Reads cover partitioned tables, v1 and v2 tables, and schema evolution (promoted, added, dropped, and renamed columns, matched by Iceberg field ID). Tables with delete files, ORC data files, or format version 3 aren't supported yet and report an error
  • The engine never deletes table files; a failed commit removes only the files it wrote

Mixing with other connectors​

A profile can hold a wvlet connector next to other engines; switch to it with use wvlet (see Working with Multiple Connectors).