Skip to main content

dbt-polars

dbt-polars integrates the flexibility and performance of Polars with the data governance and lineage capabilities of dbt.

dbt-polars is an independent project. It is not affiliated with, endorsed by, or maintained by the Polars team or dbt Labs.

Python and SQL, side by side​

Choose Python or SQL per model, whichever fits the transformation best. Both run on the same Polars engine and can freely reference each other. Unlike in dbt-core, Python models are first class and can do anything a SQL model can.

# models/daily_weather.py
import polars as pl


def model(dbt, session) -> pl.LazyFrame:
    hourly = dbt.ref("open_meteo_raw")
    return hourly.group_by("date").agg(
        pl.col("temperature_2m").min().alias("min_temperature"),
        pl.col("temperature_2m").max().alias("max_temperature"),
    )
-- models/daily_weather.sql
select
    date,
    min(temperature_2m) as min_temperature,
    max(temperature_2m) as max_temperature
from {{ ref('open_meteo_raw') }}
group by date

The flexibility of Python and Polars​

dbt-polars runs on any compute and writes to any of its catalogs

Run anywhere​

Polars executes your models in the same process as dbt. You don't need an expensive data warehouse to run your transformations. dbt-polars runs anywhere you can run Python.

Store data where you want​

dbt-polars connects to popular platforms such as Databricks, Snowflake and Microsoft Fabric. Setting one up only takes a few lines in your profile. See Connecting to a catalog for the full list.

Extensible to any platform​

Implement a custom catalog to tailor dbt-polars to your platform. A custom catalog controls authentication, where data is written and how metadata is registered. The dbt-polars engine still performs the writes. See Custom catalogs to build your own.

The familiarity of dbt​

If you know dbt, you already know how to use dbt-polars. Introduce dbt-polars to your Python projects to benefit from key dbt-features, such as lineage and documentation. See dbt compatibility for the full list of supported dbt features.

Lineage graph of a dbt-polars project

Built for efficiency​

dbt-polars builds on the full power of the Polars engine. Models run lazily, so Polars optimizes the whole query before running it. Execution and writes are streamed in batches to keep memory use low. See Performance for tips on writing efficient models.

Where to next​

  • Quick start: run your first dbt-polars project in a few minutes.
  • Demo project: a guided tutorial in which you set up a weather data processing pipeline and use many of dbt-polars' features along the way.
  • Essentials: a good starting point for using dbt-polars in your own projects, starting with connecting to a catalog.