Manual Data Load Without Calendar Date Range
  • 4 Minutes to read
  • Dark
    Light

Manual Data Load Without Calendar Date Range

  • Dark
    Light

Dataddo allows you to manually load historical data from a specific date range.

Do a one-time request in the Dataddo app and the manual data load will affect all flows and destinations connected to the source.

It is possible to do a manual data load any time after you create your source. Please keep in mind that if your destination is a data warehouse, you will need to load historical data to your data flow as well.

Calendar Date Range

Is there a calendar to choose your date range from? Refer to this article.


1. Load Historical Data to a Data Source

In some cases, it will not be possible to select your date range from a calendar like in the previous screenshot. To manually load historical data, you will need to use a date range in the source URL.

  1. Go to your source, click on three dots, and select Edit. Navigate to the Advanced Settings tab to get the URL field. In the URL, the date range can be indicated by the following:

    • "startDate" and "endDate"
    • "Since_send_time" and "before_send_time"
    • "since" and "until"
    • "start" and "end"
      Historical data load - source URL

    There are two ways to define the date range, static and dynamic.

    Static date range will allow you to extract data from a specific time period, in this example, data from September 1, 2020 until September 14, 2020 (written in the ISO 8601 time format).

    https://us16.api.mailchimp.com/3.0/campaigns?offset=0&count=500&since_send_time=2020-09-01T00:00:00+00:00&before_send_time=2020-09-14T23:59:59+00:00

    Dynamic date range extracts data relative to today’s date. In this example, we want to extract last 90 days' data, including today:

    https://us16.api.mailchimp.com/3.0/campaigns?offset=0&count=500&since_send_time={{90d0}}

  2. Once you define your date range, click on Save and proceed to loading your historical data via Manual Data Load button as in 1a.

    Historical data load - no calendar

    Choose your snapshot keeping policy depending on your destination. Append if your destination is a dashboarding app, and Replace if your destination is a data warehouse. Confirm and click on Load data.

  3. [Optional] You can check if your data extraction was successful through Show Logs and Test Extraction.

  4. If your destination is a data warehouse, load historical data to your data flow (see section 2) before proceeding to the next step.

  5. Return the URL back to its original state, otherwise in case of static date range, Dataddo will keep extracting data from this specific time period. In the case of dynamic date range, not changing the URL back will result in duplicities.

DATADDO TIP

If you chose a dynamic date range, you don’t necessarily have to change the URL back to the original. Just keep in mind that you snapshotting policy should match your dynamic date range, e.g. weekly data extraction of the last 7 days' data.
If they don’t match, depending on your snapshotting policy, this can mean that you will be extracting the last e.g. 90 days' worth of data every day.

2. Load Historical Data to a Data Flow

This section is applicable if your destination is a data warehouse. This step is very quick as it’s simply sending the data you loaded to your source to your warehouse.

Under Flows, find the one with the source you manually loaded historical data to and click on the Manual Data Load icon.
Historical data load - load data to flow

Confirm the data insert and click on Insert data. You can check in the logs or directly in your warehouse to make sure the operation was executed successfully.


Tip & Tricks for Specific Connectors

Facebook Posts

Facebook Posts data cannot be broken down by days, you can only extract the lifetime values of your metrics such as impressions or clicks. Set the snapshot keeping policy to Append to build a time series of your data.

Google Search Console

The Google Search Console API allows loading data from the past 16 months.

Google My Business

The Google My Business API allows loading data from the past 14 months.

Instagram Story

Due to the Instagram Story API limitations, Dataddo can only access data while the story is active (24-hour window). You can create a source only when you have a story currently posted to the account.

Hourly syncing frequency is recommended in order to get data as often as possible and to get the values at the end of the 24-hour window.

If you don't expect to post a story every 24 hours, in other words we need for the source to expect empty data, enable the "allow empty" option. Cick on the three dots on your source, select Edit and scroll down on the Basic Info.

Data Sources - Allow Empty

By selecting Timestamp, the data will show the metrics over time of a single story, along with the timestamp.

HubSpot

The HubSpot API does not allow selecting a date range. Dataddo will pull the full list of the selected contacts or deals.

To keep track of historical records, choose a timestamp and set the snapshot keeping policy to Append to build a time series.

TIP
To get owners' names for deals, create a new source with a specific Deal Owners dataset. Use the Data Blending feature when creating a flow, with the "owner ID" as the Join Key field.

Standard Connectors

For most of the standard connectors, you can change the date range in the URL under Advanced Settings. For Sendinblue, Simplicate, Klaviyo, and Google Search Console, dynamic data range has been implemented.


Need assistance?

Feel free to contact us and we will help you with the setup of your sources and/or flows. To speed up the process of resolving your ticket, make sure you provide us with sufficient information.


Was this article helpful?

What's Next