Connector - Wvlet Engine
The wvlet engine is an in-process columnar query engine that executes compiled Wvlet plans directly — no SQL generation and no external database. It runs on every platform the CLI supports (JVM, Node.js, and the native binary), reading Parquet, Iceberg, JSON, and CSV data through pure-Scala readers, so it also works where DuckDB is unavailable.
The engine covers queries over files and registered tables (filters, projections, joins
including ASOF joins, aggregations, window functions, subqueries, pivot, unnest, lambdas,
show commands, and TPC-H), plus save to, append to, and delete on in-memory tables and
save to / append to Parquet, CSV, and JSON files and Iceberg tables, and create table, drop table,
truncate, and create / drop schema. Features it doesn't support yet report an error:
alter table, views, update, execute, flows, table functions such as read_csv(...), and some
engine-specific functions. Map values run but can return results that
differ from DuckDB.
Usage
Select the engine with -t wvlet:
# Query a local file
wvlet run -t wvlet "from 'data/orders.parquet' where o_totalprice > 1000 limit 10"
# Run a query file
wvlet run -t wvlet -f report.wv
Relative file paths resolve against the working folder (-w).
-t wvlet without --host runs the in-process engine; with --host it connects to a
remote wvlet server instead.
Registering tables
Define a profile with a wvlet connector to reference tables by name. Its properties register
tables:
dataDir: a folder whose data files (.parquet,.json,.jsonl,.csv,.tsv, and their.gzvariants) and sub-folders (Iceberg tables or folders of Parquet files) become tables named after the file name without its extensiontables: an object mapping table names to paths
{
"profiles": [
{
"name": "local-engine",
"connectors": [
{
"name": "wvlet",
"type": "wvlet",
"properties": {
// customer.parquet, orders.parquet, ... become tables customer, orders, ...
"dataDir": "data",
"tables": {
"events": "logs/events.jsonl",
"sales": "warehouse/sales"
}
}
}
]
}
]
}
wvlet run -p local-engine "from orders join customer on o_custkey = c_custkey agg _.count"
Relative paths resolve against the working folder. The compiler sees the schemas of Parquet and Iceberg tables (read from file footers), so their column names are checked as for other engines. JSON and CSV tables are read only when a query scans them.
Saving results
save to <table> keeps a query result as a table, and append to and delete change it.
Without a warehouse, tables live in memory for the rest of the session:
from 'data/orders.parquet'
where o_orderstatus = 'F'
save to finished_orders
-- Add rows; `append to` a table that doesn't exist creates it
from 'data/orders_today.parquet'
where o_orderstatus = 'F'
append to finished_orders
-- Insert-or-update by key: rows with a matching o_orderkey update the existing rows
append to finished_orders on o_orderkey {
from 'data/order_fixes.parquet'
}
-- Remove rows
from finished_orders
where o_totalprice < 100
delete
Appended rows map to the table's columns by position, or to the columns listed in
append to t (c1, c2) (the others get their declared default, or NULL), and are cast to the
column types. A keyed
append to t on k matches columns by name, as SQL MERGE does: matched rows get the new values
of the query's columns, and rows with a NULL key always insert. Tables
registered from files (dataDir, tables) are read-only: writing to them reports an error
instead of changing the files.
Creating tables
create table makes an empty table from a table declaration, including column
defaults and not null, primary key, and unique constraints, which every write to the table
checks:
table accounts = {
id: int primary key
email: string unique not null
status: string = 'active'
}
create table accounts
-- Fills status with its default
append to accounts (id, email) {
from [[1, 'alice@example.com']] as t(id, email)
}
truncate accounts
drop table accounts
create table t if not existskeeps an existing table, andcreate or replace table tstarts over; a plaincreate tableof an existing table is an errorappend toa declared table that doesn't exist yet creates it from the declaration firstcreate schema sandsave to s.tgroup tables under a schema;drop schema s with cascade: truedrops them, anddrop schema sof a schema that still holds tables is an errordrop tableof a table registered from files only unregisters it; the files stay as they are
Changing tables
reshape (or SQL alter table) changes a table's columns, keeping its rows:
reshape accounts {
add signup_date: date -- NULL in existing rows
rename status as state
cast id as long -- change a column's type
exclude email -- drop a column
}
rename table accounts to members
SQL alter table also sets or drops a column's default and not null. Tables registered from
files can't be changed.
Persistent tables
Give the connector a warehouse folder to keep tables across sessions, as a DuckDB database file
does. Every table written by name (create table, save to t, append to t) goes there and is
read back by the next session:
{
"name": "wvlet",
"type": "wvlet",
"properties": {
"warehouse": "warehouse"
}
}
- Each table is an Iceberg table at
<warehouse>/<schema>/<table>: tables of the default schemamainare underwarehouse/main/, andcreate schema smakes thewarehouse/s/folder. DuckDB, Spark, Trino, and pyiceberg read the tables too - Every write is one atomic Iceberg commit:
save to tandtruncate treplace the rows in a new snapshot (save tomay change the columns),append to tadds a snapshot, anddeleteandappend to t on krewrite only the data files holding the rows they change (copy-on-write) reshape/alter tablechange the table's metadata only when Iceberg allows it (adding a column without a default, dropping or renaming a column, wideninginttolongor a decimal's precision, defaults, droppingnot null); other changes rewrite the table in one snapshot. Renaming a warehouse table isn't supported yet- Declared defaults and
not null/primary key/uniqueconstraints are kept in the table's properties and checked by later sessions too drop tabledeletes the table's folder, anddrop schema s with cascade: truethe schema's- A name that is both a warehouse table and a
dataDir/tablesentry is an error when the session starts. Tables registered from files stay read-only - Without
warehouse, tables are in memory only (like DuckDB's:memory:). The browser playground always keeps tables in memory
Writing files
save to '<path>' writes a query result to a file, and append to '<path>' adds rows to it
(creating the file when it doesn't exist). The extension selects the format:
| Extension | Format |
|---|---|
.parquet | Parquet |
.csv, .tsv | CSV / TSV with a header row |
.json, .jsonl, .ndjson | Newline-delimited JSON |
from 'data/orders.parquet'
where o_orderstatus = 'F'
save to 'out/finished_orders.parquet' with compression: 'zstd'
from 'data/orders_today.parquet'
where o_orderstatus = 'F'
append to 'out/finished_orders.parquet'
Options for save to ... with:
format:parquet,csv,tsv, orjson, overriding the extensioncompression(Parquet):snappy,zstd,gzip,brotli,lz4, oruncompressedheader(CSV, defaulttrue) anddelimiter(CSV)
Appended rows map to the file's columns by position. CSV, TSV, and JSON lines files get the new
rows added at the end, leaving the existing text as is; Parquet files are rewritten with the new
rows cast to the file's column types, through a temporary file so a failed append keeps the
original. Files are written to local paths only; writing to s3:// or other remote paths isn't
supported yet.
Iceberg tables
The engine reads and writes Apache Iceberg tables in local folders,
on every platform, with no Spark or DuckDB. A folder is an Iceberg table when it has a metadata/
folder holding .metadata.json files.
-- Create a table (snapshot 1), partitioned by country
from 'data/events.parquet'
save to 'warehouse/events' with format: 'iceberg', partition_by: 'country'
-- Add a snapshot
from 'data/events_today.parquet'
append to 'warehouse/events'
-- Read the current snapshot, or the one a metadata file records
from 'warehouse/events'
from 'warehouse/events/metadata/v1.metadata.json'
save to '<dir>' with format: 'iceberg'creates a table in a new or empty folder. Options:partition_by(comma-separated identity partition columns) andcompression.save toon an existing Iceberg table is an error: the engine doesn't replace Iceberg tables yet, so useappend toor write to a new folderappend to '<dir>'commits the rows as a new snapshot, mapped to the table's columns by position and cast to their types. It works on tables committed asmetadata/vN.metadata.json(as the wvlet engine and Hadoop-catalog writers create them); a table that a catalog owns, such as one written by pyiceberg or a REST catalog, is read-only.append toa folder that isn't an Iceberg table is an error, sinceappend tocan't name the format- Reads cover partitioned tables, v1 and v2 tables, and schema evolution (promoted, added, dropped, and renamed columns, matched by Iceberg field ID). Tables with delete files, ORC data files, or format version 3 aren't supported yet and report an error
- The engine never deletes table files; a failed commit removes only the files it wrote
Mixing with other connectors
A profile can hold a wvlet connector next to other engines; switch to it with use wvlet
(see Working with Multiple Connectors).