The Iceberg REST Catalog destination writes your data as Apache Iceberg tables through any catalog that implements the Iceberg REST Catalog specification. Dataddo talks to the catalog over its REST API to create and commit tables, and stores the data files in the object storage that backs the catalog.
Supported Data Format
This destination writes Apache Iceberg tables only. Dataddo manages the Iceberg metadata and data files through the catalog. There is no CSV, Parquet, or JSON file option here, because the storage layout is defined by the Iceberg table format.
Supported Catalogs
Any catalog that implements the Iceberg REST Catalog specification works with this destination. That specification is a shared REST API, so a single client can talk to many catalog backends. Catalogs that implement it include Apache Polaris, Databricks Unity Catalog, Project Nessie, Lakekeeper, AWS S3 Tables, Google BigLake, Microsoft OneLake, and AWS Glue through its Iceberg REST endpoint.
Catalogs differ in how they hand out storage credentials, and this destination supports both models:
- Credential vending. The catalog issues short-lived storage credentials for each table when Dataddo asks for it. Leave the S3 fields on the authorizer empty and let the catalog vend access.
- Direct storage credentials. The catalog does not vend credentials, so you provide the S3 access key, secret, and region on the authorizer, and Dataddo uses them to read and write the table's data files.
If you use AWS Glue as your catalog and prefer a Glue-native setup, use the dedicated AWS Glue destination instead.
Authorize Connection to Iceberg REST Catalog
Prerequisites
- A reachable Iceberg REST Catalog endpoint.
- Credentials the catalog expects (OAuth token endpoint, or username and password, or none if the catalog is open).
- Either credential vending enabled on the catalog, or S3 credentials for the object storage that holds the table data.
Create the authorizer
Go to Authorizers > Authorize New Service > Iceberg REST Catalog, then fill in the fields below and save. Only the Catalog URL is required. The other fields depend on how your catalog handles authentication and storage.
| Field | Required | Description |
|---|---|---|
| Catalog URL | Yes | The Iceberg REST Catalog endpoint, for example https://catalog.example.com. |
| Auth URL | No | The OAuth token endpoint. Leave empty if the catalog does not use OAuth. |
| Username | No | The username for catalog authentication. Leave empty if not required. |
| Password | No | The password or token for catalog authentication. |
| Warehouse | No | The warehouse location, for example s3://my-bucket/warehouse. Leave empty if the catalog sets it server side. |
| S3 Access Key ID | No | The S3 access key ID for storage access. Required only if the catalog does not vend credentials. |
| S3 Secret Access Key | No | The S3 secret access key. Required only if the catalog does not vend credentials. |
| S3 Region | No | The AWS region for the S3 storage. Required only if you provide S3 credentials. |
Create an Iceberg REST Catalog Destination
Go to Destinations > Create Destination > Iceberg REST Catalog, choose the authorizer you created, then set the fields below.
| Field | Description |
|---|---|
| Destination Name | A name for this destination in Dataddo. |
| Namespace / Database | The Iceberg namespace (database) that holds your tables. The value cannot contain whitespace. |
Write Modes
Write mode decides how each run changes the target table. The available modes are resolved by Dataddo's writer at flow time, so the exact list can depend on the catalog and the current table state. The flow default is truncate_insert.
| Mode | Meaning |
|---|---|
insert |
Appends the run's rows to the table. |
truncate_insert |
Empties the table, then writes the current data. |
Table Naming
The Table field on the flow sets the table name. Dataddo writes to namespace.table, where the namespace comes from the destination and the table name comes from the flow. The table is created in the namespace on first write when it does not exist yet.
How to Create a Flow to Iceberg REST Catalog
- Go to Flows > Create Flow.
- Add one or more sources.
- Add the Iceberg REST Catalog destination.
- Choose the write mode.
- Enter the table name.
- Set the schedule and save.
Troubleshooting
- Namespace rejected. The namespace cannot contain whitespace. Use a single token such as
my_namespace. - Catalog unreachable or unauthorized. Confirm the Catalog URL, and the OAuth or username and password the catalog expects.
- Write fails on storage. If your catalog does not vend credentials, confirm the S3 access key, secret, and region on the authorizer, and that they can write to the warehouse path.