Dataddo

Dataddo is a cloud-based data integration platform that enables businesses to connect and automate data workflows across various sources. It allows users to extract, transform, and load (ETL) data from different platforms into a centralized location, facilitating efficient data management and analysis.

The Dataddo connector is different from other connectors: it extracts metadata about your own Dataddo account. It exposes your sources, flows, authorizers, and execution logs as plain tables. Export them to a dashboard, database, or data warehouse to build data governance and pipeline monitoring views, and get a complete picture of how your Dataddo pipelines perform.

Authorize Connection to Dataddo

1. On the Authorizers page, click on Authorize New Service and select Dataddo.
2. Fill in the following fields:
   1. Email: Email address you use to log in to Dataddo.
   2. Password: Password to your Dataddo account.
3. Rename your authorizer for easier identification and click on Save.

If you created your account using an SSO provider, we recommend creating an additional system user account with email and password as login credentials. For more information, see our guide on adding a team member.

Data Coverage

The connector exposes five datasets. Together they describe your whole account: which pipelines exist, how they are wired together, and how every run went.

The sections below list every column each dataset returns.

Source Metadata

One row for every data source in your account.

Use it for: pipeline inventory, finding unused sources (flow_count is 0), and freshness checks (last versus next execution).

Flows Metadata

One row for every flow in your account.

Use it for: destination coverage, flow schedules, and write freshness.

Source association to Flows

One row for every link between a source and a flow. This is the mapping table that connects Source Metadata and Flows Metadata.

Use it for: joining sources to flows and building lineage views (source to flow to destination).

Authorizers Metadata

One row for every authorizer, that is, every stored service credential in your account.

Use it for: credential governance. See which credentials exist, what state they are in, and when they were added.

Execution logs

One row for every job run: extractions (source runs) and writes (flow runs).

Use it for: run health, error monitoring, throughput (rows per run), and run duration (finished_at minus started_at).

How the Datasets Join Together

All joins are plain equality joins on ID columns:

Chaining Source Metadata, Source association to Flows, and Flows Metadata gives you full lineage: one row per source and destination pair, from the source through the flow to destination_type and destination_id.

How Data Extraction Works

What each run pulls depends only on whether the dataset supports a date range (see the Date range column above):

• Date range supported (Yes): every run extracts the records inside the selected relative window, for example "Last 24 Hours". The window slides forward with the current date. For Execution logs the window filters on when the log entry was created. For Source Metadata and Flows Metadata it filters on when the source or flow was created, so pick the widest range if you want a complete inventory. Older records need a full data re-sync with a wider range. See Data Backfilling.
• No date range (No): every run pulls all available data.

Set the relative date range when you create the source.

Recommended Configurations

The setups below cover the most common use cases. For a full governance dashboard, create one Dataddo source per dataset and join them in your BI tool or data warehouse.

Pipeline health monitoring

Track failed and slow runs across your account.

• Dataset: Execution logs.
• Date range: Last 24 Hours. Schedule the source to run hourly.
• Enable dataddo_hash and write with upsert, so overlapping windows do not create duplicate rows.
• Dashboard ideas: runs by status, error messages of failed runs, total_rows over time, run duration (finished_at minus started_at).

Data governance catalog

Build a live inventory of all pipelines and credentials.

• Datasets: Source Metadata, Flows Metadata, Source association to Flows, and Authorizers Metadata (four sources).
• Date range: the widest available for Source Metadata and Flows Metadata, because their range filters on the creation date.
• Schedule daily. Enable dataddo_hash and write with upsert to keep one current row per object.
• Join the tables as shown above. Dashboard ideas: sources by type, flows by destination type, unused sources (flow_count is 0), authorizers by status.

Freshness and schedule monitoring

Catch pipelines that stopped running.

• Datasets: Source Metadata and Flows Metadata.
• Enable dataddo_extraction_timestamp so every row records when the snapshot was taken.
• Alert when Last Execution is older than you expect, or when Next Scheduled Execution has passed without a new run.

Account history over time

Chart how your account grows and changes.

• Datasets: any of the metadata datasets.
• Enable dataddo_extraction_timestamp and write with insert (append). Each sync adds a dated snapshot.
• Use the snapshots to chart the number of sources and flows over time, or to build slowly changing dimensions.

Metadata Columns

When you create a source, you can add these Dataddo metadata columns to the extracted data:

• dataddo_hash - a fingerprint built from each record's key fields. It works as a natural key, so it is ideal for upserts (updating existing rows in your destination instead of creating duplicates).
• dataddo_extraction_timestamp - the date and time the row was extracted. Use it to track how records change over time, for example to build slowly changing dimensions.

How to Create a Dataddo Data Source

Creating a data source takes you through six steps, shown in the progress bar at the top of the wizard. Each step is explained below.

1. Pick the connector

On the Sources page, click Create Source, then select the connector from the catalog. Use the search bar or the category tabs if you do not see it right away. You can rename the source at any time using the pencil icon next to its name.

2. Select the dataset

A dataset defines the shape of your data: which fields you get and how they relate. Select the dataset you want; you can still fine-tune the exact fields later.

• Each dataset has a short description of what it contains. Use the search box to find a dataset, attribute, or metric by name.
• The panel on the right previews the selected dataset's fields. For each field you can see its data type, whether it holds sensitive data (personal fields such as name or email are flagged), and which other datasets it links to, so you can see how the datasets relate.

3. Choose the account

This step selects what Dataddo reads from.

• Authorizer: Select an account you have already authorized from the drop-down. If you have none yet, choose Add new account and follow the prompts. If no authorizer is selected, Dataddo asks you to authorize before you continue.
• What to extract from: Select the exact entity you want to pull data from. Depending on the service this may be labelled an account, property, profile, workspace, or similar, sometimes with a sub-level to choose as well.
• Multiple accounts: To pull the same data from every entity you can access, turn on Automatically collect data from all .... This is multi-account extraction. Leave it off to choose them by hand.

4. Refine the attributes and metrics

The dataset already sets the structure. Here you fine-tune it: tick or untick the specific attributes and metrics you want to keep, and use the search box to find a field quickly. Click Test on Sample Data at any point to preview the result before you continue.

5. Add metadata columns (optional)

Two optional columns help your destination handle the data.

• Dataddo Hash (Include Row Hash): a fingerprint built from the columns you pick. It works as a natural key, so your destination can deduplicate rows and run upserts instead of creating duplicates. Turn it on, then select the columns that uniquely identify a row.
• Dataddo Extraction Timestamp: the time each row was extracted. Use it to watermark the data, for example to build slowly changing dimensions or to track when a value last changed.

6. Set the schedule

Decide how often Dataddo runs the extraction.

• Frequency: how often the pipeline runs, for example daily. Click Show advanced settings to also set the exact hour and minute (UTC).
• Date range: the relative window each run extracts, for example "Yesterday". The window moves forward on every run.
• Historical data: a new source starts from the current window. To load older data, run a full data re-sync after the source is created.
• Allow Empty Data Extractions: when on, a run that returns no data records zero rows instead of failing. Turn it on if the source can legitimately have periods with no data.

Click Save. Your data source is ready.

Troubleshooting

Data Preview Unavailable

No data preview when you click on Test Data might be caused by an issue with your source configuration. The most common causes are:

• Date range: Try a smaller date range. You can load the rest of your data afterward via manual data load.
• Insufficient permissions: Please make sure your authorized account has at least admin-level permissions.

Related Articles

Now that you have successfully created a data source, see how you can connect your data to a dashboarding app or a data storage.

Sending Data to Dashboarding Apps

• Simple Data Integration to Dashboards
• Data Backfilling to Dashboarding Apps

Sending Data to Data Storages

• Batch Ingestion to Data Warehouses
• Data Backfilling to Storages

Other Resources

• Troubleshooting
• Extraction Logs
• Data Duplication