DuckDB Data Accelerator
To use DuckDB as Data Accelerator, specify duckdb
as the engine
for acceleration.
datasets:
- from: spiceai:path.to.my_dataset
name: my_dataset
acceleration:
engine: duckdb
Configuration​
The DuckDB accelerator can be configured by providing the following params
:
duckdb_file
: The name for the file to back the DuckDB database. If the file does not exist, it will be created. Only applies ifmode
isfile
.
Configuration params
are provided in the acceleration
section for a data store. Other common acceleration
fields can be configured for DuckDB, see see datasets.
datasets:
- from: spiceai:path.to.my_dataset
name: my_dataset
acceleration:
engine: duckdb
mode: file
params:
duckdb_file: /my/chosen/location/duckdb.db
Limitations
- The DuckDB accelerator does not support schemas with field types: nested arrays/lists, UTF8/string arrays/lists, structs, nested structs or map fields. For example:
- Supported:
SELECT [1, 2, 3];
SELECT ['1992-09-20 11:30:00.123456789', 'epoch'::TIMESTAMP]
- Unsupported:
SELECT [['duck', 'goose', 'heron'], ['frog', 'toad']]
SELECT {'x': 1, 'y': 2, 'z': 3}
SELECT MAP(['key1', 'key2', 'key3'], [10, 20, 30])
SELECT ['duck', 'goose', 'heron'];
- Supported:
- The DuckDB accelerator does not support
Decimal256
(76 digits), as it exceeds DuckDB's maximum Decimal width of 38 digits. - The DuckDB accelerator does not support using the same
duckdb_file
for multiple accelerated datasets. - Updating a dataset with DuckDB acceleration while the Spice Runtime is running (hot-reload) will cause DuckDB accelerator query federation to disable until the Runtime is restarted.