Iceberg (generic)

Store tables in Apache Iceberg format using pyiceberg.

info

For Iceberg tables in Databricks, Snowflake or Microsoft Fabric, use the dedicated Databricks, Snowflake or Fabric catalog instead.

Supported features

Feature	Status	Remarks
Read/write Iceberg tables	✅	Unsigned integer columns are written as signed, since Iceberg has no unsigned types
All catalogs supported by pyiceberg	✅	E.g. REST, SQL, Hive and Glue, see pyiceberg's catalogs

Installation

pip install 'dbt-polars[iceberg]'

Profile configuration

Example

A fully local Iceberg setup with a SQLite catalog and a local warehouse:

Single catalog

my_project:
  target: dev
  outputs:
    dev:
      type: polars
      catalog_type: iceberg
      schema: dev
      uri: sqlite:///catalog.db         # SQLite database for the Iceberg metastore
      warehouse: file://warehouse       # directory where table data is written

Multiple catalogs

my_project:
  target: dev
  outputs:
    dev:
      type: polars
      catalog: my_iceberg
      catalogs:
        - name: my_iceberg
          type: iceberg
          schema: dev
          uri: sqlite:///catalog.db     # SQLite database for the Iceberg metastore
          warehouse: file://warehouse   # directory where table data is written
        - name: other_catalog
          ...

See Combining catalogs.

A REST catalog, such as Databricks Unity Catalog:

    dev:
      type: polars
      catalog_type: iceberg
      schema: dev
      pyiceberg_type: rest
      uri: https://<workspace>.azuredatabricks.net/api/2.1/unity-catalog/iceberg-rest/
      token: <databricks-pat-or-token>
      warehouse: <unity-catalog-name>    # the Unity Catalog catalog name (e.g. "main")

Parameter Reference

The following parameters are supported:

• catalog_type - (Required) Must be iceberg.

• schema - (Required) Default schema for models. dbt-polars creates missing schemas.

• pyiceberg_type - (Optional) Sets the pyiceberg type property, e.g. rest, hive or glue. It is a separate key because type is already used by dbt-polars.

• All other parameters - (Optional) Passed to pyiceberg.catalog.load_catalog, see the pyiceberg docs for the options of each backend.