The Databricks Lakehouse Platform combines the best of data lakes and data warehouses, simplifying the modern data stack and eliminating data silos. Built on open standards and open source, the platform provides a common approach to data management, security, and governance, enabling businesses to operate more efficiently, innovate faster, and achieve the full potential of their analytics and AI initiatives.
Authorize Connection to Databricks
Prerequisites
- A running Databricks instance reachable by Dataddo. If you use a firewall, allow the Dataddo IPs.
- A database user with rights to create tables and to insert, update, and delete rows in the target schema.
Create the authorizer
In Authorizers, click Authorize New Service and select Databricks, then fill in the fields:
Databricks SQL Warehouse
| Field | Description |
|---|---|
| A name for this authorizer in Dataddo | A label so you can recognize the connection later. |
| Host | Host address of your Databricks SQL Warehouse |
| Port | Port number for the connection. Default is 443 for HTTPS. |
| Token | Your Databricks Personal Access Token (PAT). |
| Warehouse | SQL Warehouse ID. Found in your Databricks SQL Warehouse connection details. |
Databricks OAuth
| Field | Description |
|---|---|
| A name for this authorizer in Dataddo | A label so you can recognize the connection later. |
| Hostname or workspace URL | Hostname or workspace URL |
| HTTP Path | HTTP Path |
| Redirect Uri | Fill only for U2M (Per-User) connection Default: https://app.dataddo.com/settings/service/databricksOauth. |
| Client ID | Client ID |
| Client Secret | Client Secret |
Click Save. Dataddo validates the connection.
Create a Databricks Destination
Go to Destinations, click Create Destination, and select Databricks. Give the destination a name, choose the authorizer, then set:
| Field | Description |
|---|---|
| Catalog | Select the Unity Catalog you would like to load data to. Shown when oAuthId is ``. |
| Schema | Select the schema within the chosen catalog that you would like to load data to. Shown when catalog is ``. |
Click Save.
Write Modes
The write mode decides how each flow run changes the target table. Databricks offers the standard set below, and the default is insert. Which modes are available, and whether you can switch modes on an existing flow, can depend on the table's current state.
| Mode | Meaning | Composite Key |
|---|---|---|
insert |
Appends new rows on every run. | No |
insert_ignore |
Appends rows, skipping rows that already exist. | Yes |
truncate_insert |
Empties the table, then writes the current data. | No |
upsert |
Inserts new rows and updates existing rows matched by the Composite Key. | Yes |
update |
Updates existing rows matched by the Composite Key; fails rows with no match. | Yes |
update_ignore |
Updates matched rows, skipping rows with no match. | Yes |
delete |
Deletes rows matched by the Composite Key. | Yes |
Composite Key
The Composite Key is the set of columns whose combined values identify each row. Every mode except insert and truncate_insert needs one. You can select up to 4 columns.
Table Naming
Dataddo creates the table on the first write if it does not exist. When you name the table:
- Table name: table name should contains only letters, numbers or underscores
- You can put date-range patterns such as
{{1d1}}in the table name.
Loading Files into Databricks Storage
This destination ingests data straight into Unity Catalog tables in your SQL Warehouse. If instead you want the data to land as files in the cloud storage that backs your lakehouse (ADLS, Azure Blob, or Amazon S3), use the matching file destination and point it at that storage:
Those destinations write CSV, JSON, JSONL, or Parquet files, which you can then read from Databricks as external tables or Unity Catalog volumes. See File Partitioning on each file destination for date-based layouts a lakehouse can read as partitions.
How to Create a Flow to Databricks
- Go to Flows and click Create Flow.
- Add one or more sources.
- Add Databricks as the destination and pick the authorizer.
- Choose a write mode, and a Composite Key for modes that need one.
- Set the schema and the table name.
- Set the schedule and click Save.
Troubleshooting
Invalid schema or table name
The schema or table name does not match Databricks's naming rules. Fix the name in the flow to match the rules in Table Naming, then restart the flow.
Insufficient privileges
The database user cannot create tables or write rows in the target schema. Grant the user create, insert, update, and delete rights, then restart the flow.
Cannot connect to Databricks
Dataddo cannot reach Databricks. Check the credentials in the authorizer, and make sure the server is reachable by Dataddo. If you use a firewall, allow the Dataddo IPs.