Documentation Index

Fetch the complete documentation index at: https://docs.dataddo.com/llms.txt

Use this file to discover all available pages before exploring further.

Databricks as a Source

  • 4 minute(s) read
Prev Next

Databricks is a cloud-based lakehouse platform that unifies data warehousing and analytics on top of Apache Spark.

Authorize Connection to Databricks

Before you can create a data source, authorize Dataddo to connect to your Databricks database.

Prerequisites

  • A running Databricks instance reachable from the internet via a public IP or hostname.
  • A database user with read access to the tables you want to extract.
  • Your firewall configured to allow Dataddo's IP addresses. See Network ACL.

Create the authorizer

  1. In Dataddo, go to Authorizers and click Authorize New Service.
  2. Select Databricks SQL Warehouse.
  3. Fill in the connection details:
Field Details
Label A name for this authorizer in Dataddo.
Host Host address of your Databricks SQL Warehouse
Port Port number for the connection. Default is 443 for HTTPS.
Token Your Databricks Personal Access Token (PAT).
Warehouse SQL Warehouse ID. Found in your Databricks SQL Warehouse connection details.
  1. Click Save. Dataddo validates the connection before the authorizer is created.

Capabilities

Databricks supports the data extraction methods below. Each method differs in which row changes it captures and how much load it places on the source. Choose the one that matches your table and use case; for a full explanation and setup detail, see Database Replication.

Method New rows Updated rows Deleted rows Details
Table Replication by Timestamp Yes Yes No Tracks a datetime column (such as updated_at) and re-extracts a row whenever that timestamp advances. Low to medium load on the source.
Table Replication by Row Sequence Yes No No Tracks a continuously increasing numeric column (such as an auto-increment ID) to capture inserts only. Low load on the source, and supports an optional per-run row limit for very large tables.
Log-based Replication (CDC) - - - Not available for this connector.
Custom SQL Query Depends on query Depends on query Depends on query Runs your own SQL query, so you control exactly which rows and columns are returned, including joins, filters, and aggregation.

How the Incremental Methods Track Changes

The two incremental methods track progress differently:

  • Table Replication by Timestamp extracts the rows whose Change Tracking Column falls inside the source's relative date range (for example "last 24 hours"), and that window moves forward with the current date. A row re-enters the window whenever its timestamp is refreshed, which is how updates are captured. Two practical consequences: the tracking column must be set on insert and refreshed on every update (such as updated_at), and the window must be at least as wide as the gap between two runs, otherwise rows changed in between are missed.
  • Table Replication by Row Sequence remembers the highest value of the Sequence Tracking Column(s) extracted so far, and each run continues from that value. The first run starts from the beginning of the table. The optional Row limit per run caps one run's size, so the initial load of a very large table can be split across several scheduled runs.
  • In both methods, your optional WHERE clause is combined with the automatic tracking filter, so do not repeat the time or sequence condition in it.

How to Create a Databricks Data Source

  1. In Dataddo, go to Sources > Create Source and select the Databricks connector.
  2. Choose the Authorizer you created above.
  3. Select the extraction method that fits your use case (see the Capabilities table above).
  4. Fill in the required fields for the selected method, such as the schema, table, and columns.
  5. For the incremental methods, select the tracking column: the Change Tracking Column (a datetime column such as updated_at) for Table Replication by Timestamp, or the Sequence Tracking Column(s) (a strictly increasing numeric column such as an auto-increment ID) for Table Replication by Row Sequence.
  6. (Optional) Add a WHERE clause to filter the extracted rows. Do not repeat the time or sequence condition in it; Dataddo adds the tracking filter automatically.
  7. Click Test Data to preview the result, then click Save.

Troubleshooting

The OAuth refresh token is invalid. The run reports Refresh token is invalid with POST .../oidc/v1/token -> 400 Bad Request.

  • Cause: the stored OAuth refresh token was invalidated, typically after the user revoked access, changed their Databricks password, or the workspace was reconfigured.
  • Fix: re-authorize the Databricks connector in Dataddo. If using a personal access token, generate a new one (User Settings, Developer, Access tokens) and update it in Dataddo.

A column is missing or the source fails on an unsupported data type. Cast the column to a supported type using a Custom SQL Query, for example SELECT CAST(my_column AS VARCHAR) AS my_column FROM my_table.

The data preview is empty. This is usually caused by one of the following:

  • the selected date range contains no data;
  • the database user does not have permission to read the selected table;
  • the selected table, columns, or tracking column are no longer valid;
  • an incompatible combination of options was selected.

Related Articles