Manual Data Load Without Calendar Date Range

Dataddo allows you to manually load historical data from a specific date range.

Do a one-time request in the Dataddo app and the manual data load will affect all flows and destinations connected to the source.

It is possible to do a manual data load any time after you create your source. Please keep in mind that if your destination is a data warehouse, you will need to load historical data to your data flow as well.

1. Load Historical Data to a Data Source

In some cases, it will not be possible to select your date range from a calendar like in the previous screenshot. To manually load historical data, you will need to use a date range in the source URL.

1. Go to your source, click on three dots, and select Edit. Navigate to the Advanced Settings tab to get the URL field. In the URL, the date range can be indicated by the following:

   • "startDate" and "endDate"
   • "Since_send_time" and "before_send_time"
   • "since" and "until"
   • "start" and "end"
     

   There are two ways to define the date range, static and dynamic.

   Static date range will allow you to extract data from a specific time period, in this example, data from September 1, 2020 until September 14, 2020 (written in the ISO 8601 time format).

   https://us16.api.mailchimp.com/3.0/campaigns?offset=0&count=500&since_send_time=2020-09-01T00:00:00+00:00&before_send_time=2020-09-14T23:59:59+00:00

   Dynamic date range extracts data relative to today’s date. In this example, we want to extract last 90 days' data, including today:

   https://us16.api.mailchimp.com/3.0/campaigns?offset=0&count=500&since_send_time={{90d0}}

2. Once you define your date range, click on Save and proceed to loading your historical data via Manual Data Load button as in 1a.

   

   Choose your snapshot keeping policy depending on your destination. Append if your destination is a dashboarding app, and Replace if your destination is a data warehouse. Confirm and click on Load data.

3. [Optional] You can check if your data extraction was successful through Show Logs and Test Extraction.

4. If your destination is a data warehouse, load historical data to your data flow (see section 2) before proceeding to the next step.

5. Return the URL back to its original state, otherwise in case of static date range, Dataddo will keep extracting data from this specific time period. In the case of dynamic date range, not changing the URL back will result in duplicities.

2. Load Historical Data to a Data Flow

This section is applicable if your destination is a data warehouse. This step is very quick as it’s simply sending the data you loaded to your source to your warehouse.

Under Flows, find the one with the source you manually loaded historical data to and click on the Manual Data Load icon.

Confirm the data insert and click on Insert data. You can check in the logs or directly in your warehouse to make sure the operation was executed successfully.

Tip & Tricks for Specific Connectors

Facebook Posts

Facebook Posts data cannot be broken down by days, you can only extract the lifetime values of your metrics such as impressions or clicks. Set the snapshot keeping policy to Append to build a time series of your data.

Google Search Console

The Google Search Console API allows loading data from the past 16 months.

Google My Business

The Google My Business API allows loading data from the past 14 months.

Instagram Story

Due to the Instagram Story API limitations, Dataddo can only access data while the story is active (24-hour window). You can create a source only when you have a story currently posted to the account.

Hourly syncing frequency is recommended in order to get data as often as possible and to get the values at the end of the 24-hour window.

If you don't expect to post a story every 24 hours, in other words we need for the source to expect empty data, enable the "allow empty" option. Cick on the three dots on your source, select Edit and scroll down on the Basic Info.

By selecting Timestamp, the data will show the metrics over time of a single story, along with the timestamp.

HubSpot

The HubSpot API does not allow selecting a date range. Dataddo will pull the full list of the selected contacts or deals.

To keep track of historical records, choose a timestamp and set the snapshot keeping policy to Append to build a time series.

Standard Connectors

For most of the standard connectors, you can change the date range in the URL under Advanced Settings. For Sendinblue, Simplicate, Klaviyo, and Google Search Console, dynamic data range has been implemented.