dbt feature compatibility
Overview of features dbt-polars already supports and plans to support in the future.
| Status | Meaning |
|---|---|
| ✅ | Supported |
| 🟡 | Partially supported |
| 🚧 | Not supported yet, but planned |
| ❌️ | Not supported, unplanned |
dbt core features
| Feature | Status | Remarks |
|---|---|---|
| table materialization | ✅ | |
| incremental materialization | 🟡 | ✅ append ✅ merge ✅ delete+insert 🚧 microbatch |
| ephemeral materialization | ✅ | |
| view materialization | ❌️ | dbt-polars has no database to hold views. Use table or ephemeral instead. |
| Snapshots | ✅ | |
| Seeds | ✅ | |
| Sources | ✅ | |
| Python models | ✅ | |
| Data tests (generic and singular SQL) | ✅ | |
| Python singular tests | ✅ | |
| Unit tests (defined in YAML) | ✅ | |
| persist_docs | ✅ | |
| dbt docs generate | ✅ | |
| dbt show | ✅ | |
| dbt clone | ❌️ | |
| Hooks | ❌️ | |
| Grants | ❌️ | |
| Model contracts | 🚧 |
dbt core extensions
dbt-polars adds three features that are not supported in dbt-core:
- Ephemeral Python models
- Python data tests
- Unit tests on Python models
Macros
dbt-core publishes a list of cross-database macros that each warehouse should implement.
| Macro group | Status | Remarks |
|---|---|---|
| Date/time | ✅ | |
| Hash | ❌️ | |
| Type conversion | ✅ | |
| Strings | ✅ | |
| Arrays | 🟡 | Missing: - array_append - array_concat |
| Aggregates | 🟡 | Missing: listagg with an order or limit |
| Set operations | ✅ | |
| Other | ✅ |
Unsupported macros raise a compilation error. Use a Python model for the same logic.
dbt packages
The table counts how many functions (macros and generic tests) of popular dbt packages work on dbt-polars.
| Package | Tested version | Supported | Partially supported | Not supported |
|---|---|---|---|---|
| dbt_utils | 1.4.1 | 39 | 0 | 5 |
| codegen | 0.14.1 | 4 | 0 | 1 |
| audit_helper | 0.11.0 | 5 | 1 | 1 |
| dbt_expectations | 0.10.10 | 58 | 0 | 6 |
dbt_utils
Not supported:
generate_surrogate_keyneeds a hash function, which Polars SQL doesn't have.get_relations_by_patternandget_relations_by_prefixqueryinformation_schema, which dbt-polars doesn't provide.at_least_oneuses awithclause inside a subquery andhavingwithoutgroup by. Polars SQL supports neither.not_constantuseshavingwithoutgroup by. It works when you setgroup_by_columns.
codegen
Not supported:
generate_sourcelists tables throughinformation_schema. It works when you passtable_names.
audit_helper
Not supported:
compare_all_columnsuses awithclause inside a subquery, which Polars SQL doesn't support.
Partially supported:
compare_relation_columnsreturns an emptycolumn_namefor columns that only exist inb_relation.
Primary keys made of several columns don't work in any macro, because combining them needs a hash function.
dbt_expectations
The regex tests (expect_column_values_to_match_regex, expect_column_values_to_not_match_regex and their _list variants) need a dbt-polars version of a dbt_expectations macro. Add this to your dbt_project.yml to use it:
dispatch:
- macro_namespace: dbt_expectations
search_order: ["dbt", "dbt_expectations"]
Not supported:
expect_column_median_to_be_betweenandexpect_column_quantile_values_to_be_betweenusepercentile_cont, which Polars SQL doesn't have.expect_row_values_to_have_recent_dataandexpect_grouped_row_values_to_have_recent_datause dbt_date's current time, which needsconvert_timezone.expect_column_values_to_be_within_n_moving_stdevsuses arows betweenwindow frame, which Polars SQL doesn't support.expect_select_column_values_to_be_unique_within_recordneeds a hash function.