Skip to main content

Data Accelerators

Data sourced by Data Connectors can be locally materialized and accelerated using a Data Accelerator.

Acceleration is enabled on a dataset by setting the acceleration configuration. E.g.

- name: accelerated_dataset
enabled: true

For the complete reference specification see datasets.

By default, datasets will be locally materialized using in-memory Arrow records.

Data Accelerators using DuckDB, SQLite, or PostgreSQL engines can be used to materialize data in files or attached databases.

Currently supported Data Accelerators include:

Engine NameDescriptionStatusEngine Modes
arrowIn-Memory Arrow RecordsAlphamemory
duckdbEmbedded DuckDBAlphamemory, file
sqliteEmbedded SQLiteAlphamemory, file
postgresAttached PostgreSQLAlpha

Data types

Data accelerators may not support all possible Apache Arrow data types. For complete compatibility, see specifications.

Refresh SQL

For datasets configured with a full refresh mode, this is an optional setting that filters the locally accelerated data to a smaller working set. This can be useful if your application/dashboard only ever uses a subset of the data stored in the federated table.

Filters will be pushed down to the remote source, and only the requested data will be transferred over the network.

- name: accelerated_dataset
enabled: true
refresh_mode: full
refresh_interval: 10m
refresh_sql: |
SELECT * FROM accelerated_dataset WHERE city = 'Seattle'

For the complete reference, view the refresh_sql section of datasets.

  • The refresh SQL only supports filtering data from the current dataset - joining across other datasets is not supported.
  • Selecting a subset of columns isn't supported - the refresh SQL needs to start with SELECT * FROM {name}.
  • Queries for data that have been filtered out will not fall back to querying against the federated table.

Data Accelerator Docs