Documentation Index

Fetch the complete documentation index at: https://docs.dataddo.com/llms.txt

Use this file to discover all available pages before exploring further.

Google BigQuery as a Source

  • 4 minute(s) read
Prev Next

Google BigQuery is a fully managed, serverless cloud data warehouse for large-scale analytics.

Authorize Connection to Google BigQuery

Before you can create a data source, authorize Dataddo to connect to your Google BigQuery database.

Prerequisites

  • A running Google BigQuery instance reachable from the internet via a public IP or hostname.
  • A database user with read access to the tables you want to extract.
  • Your firewall configured to allow Dataddo's IP addresses. See Network ACL.

Create the authorizer

  1. In Dataddo, go to Authorizers and click Authorize New Service.
  2. Select Google BigQuery service account.
  3. Fill in the connection details:
Field Details
Label A name for this authorizer in Dataddo.
Configuration file Upload the JSON configuration file, which you have downloaded from your Google Cloud account, to Dataddo. Navigate to Certificates & Tokens to complete this step.
  1. Click Save. Dataddo validates the connection before the authorizer is created.

Capabilities

Google BigQuery supports the data extraction methods below. Each method differs in which row changes it captures and how much load it places on the source. Choose the one that matches your table and use case; for a full explanation and setup detail, see Database Replication.

Method New rows Updated rows Deleted rows Details
Table Replication by Timestamp Yes Yes No Tracks a datetime column (such as updated_at) and re-extracts a row whenever that timestamp advances. Low to medium load on the source.
Table Replication by Row Sequence Yes No No Tracks a continuously increasing numeric column (such as an auto-increment ID) to capture inserts only. Reads through the BigQuery Storage Read API; a per-run row limit is not available.
Log-based Replication (CDC) - - - Not available for this connector.
Custom SQL Query Depends on query Depends on query Depends on query Runs your own SQL query, so you control exactly which rows and columns are returned, including joins, filters, and aggregation.

How the Incremental Methods Track Changes

The two incremental methods track progress differently:

  • Table Replication by Timestamp extracts the rows whose Change Tracking Column falls inside the source's relative date range (for example "last 24 hours"), and that window moves forward with the current date. A row re-enters the window whenever its timestamp is refreshed, which is how updates are captured. Two practical consequences: the tracking column must be set on insert and refreshed on every update (such as updated_at), and the window must be at least as wide as the gap between two runs, otherwise rows changed in between are missed.
  • Table Replication by Row Sequence remembers the highest value of the Sequence Tracking Column(s) extracted so far, and each run continues from that value. The first run starts from the beginning of the table. Reads go through the BigQuery Storage Read API, so a per-run row limit is not available for this connector.
  • In both methods, your optional WHERE clause is combined with the automatic tracking filter, so do not repeat the time or sequence condition in it.

How to Create a Google BigQuery Data Source

  1. In Dataddo, go to Sources > Create Source and select the Google BigQuery connector.
  2. Choose the Authorizer you created above.
  3. Select the extraction method that fits your use case (see the Capabilities table above).
  4. Fill in the required fields for the selected method, such as the schema, table, and columns.
  5. For the incremental methods, select the tracking column: the Change Tracking Column (a datetime column such as updated_at) for Table Replication by Timestamp, or the Sequence Tracking Column(s) (a strictly increasing numeric column such as an auto-increment ID) for Table Replication by Row Sequence.
  6. (Optional) Add a WHERE clause to filter the extracted rows. Do not repeat the time or sequence condition in it; Dataddo adds the tracking filter automatically.
  7. Click Test Data to preview the result, then click Save.

Troubleshooting

The OAuth token expired or was revoked. The run reports invalid_grant - Token has been expired or revoked or oauth2: "invalid_grant" "Bad Request".

  • Cause: the OAuth refresh token stored in Dataddo is no longer valid, for example after a Google password change, the user revoking the Dataddo app, or the authorizing account being deleted.
  • Fix: open the BigQuery connector in Dataddo, click Re-authorize, and complete the Google OAuth flow with the account that owns the project. For a service account, re-upload the JSON key file.

A column is missing or the source fails on an unsupported data type. Cast the column to a supported type using a Custom SQL Query, for example SELECT CAST(my_column AS VARCHAR) AS my_column FROM my_table.

The data preview is empty. This is usually caused by one of the following:

  • the selected date range contains no data;
  • the database user does not have permission to read the selected table;
  • the selected table, columns, or tracking column are no longer valid;
  • an incompatible combination of options was selected.

Related Articles