DuckDB
Overview
DuckDB is an in-process SQL OLAP database management system and this destination is meant to use locally if you have multiple smaller sources such as GitHub repos, some social media and local CSVs or files you want to run analytics workloads on. This destination writes data to the MotherDuck service, or to a file on the local filesystem on the host running Airbyte.
Use with MotherDuck
This DuckDB destination is compatible with MotherDuck.
Specifying a MotherDuck Database
To specify a MotherDuck-hosted database as your destination, simply provide your database uri with the normal md: database prefix in the destination_path configuration option.
We do not recommend providing your API token in the md: connection string, as this may cause your token to be printed to execution logs. Please use the MotherDuck API Key setting instead.
Authenticating to MotherDuck
For authentication, you can can provide your MotherDuck Service Credential as the motherduck_api_key configuration option.
Sync Overview
Output schema
Each table will contain 3 columns:
_airbyte_ab_id: a uuid assigned by Airbyte to each event that is processed._airbyte_emitted_at: a timestamp representing when the event was pulled from the data source._airbyte_data: a json blob representing with the event data.
Features
| Feature | Supported | |
|---|---|---|
| Full Refresh Sync | Yes | |
| Incremental - Append Sync | Yes | |
| Incremental - Append + Deduped | No | |
| Namespaces | No |
Performance consideration
This integration will be constrained by the speed at which your filesystem accepts writes.
Troubleshooting
Error message Request failed: (UNAVAILABLE, RPC 'GET_WELCOME_PACK')
This error may indicate that you are connecting with a 0.10.x DuckDB client (as per DuckDB Destination connector versions >=0.4.0) and your database has not yet been upgraded to a version >=0.10.x. To resolve this, you'll need to manually upgrade your database or revert to a previous version of the DuckDB Destination connector.