- 4 Minutes to read
-
DarkLight
Manual Data Load Without Calendar Date Range
- 4 Minutes to read
-
DarkLight
Dataddo allows you to manually load historical data from a specific date range.
Do a one-time request in the Dataddo app and the manual data load will affect all flows and destinations connected to the source.
It is possible to do a manual data load any time after you create your source. Please keep in mind that if your destination is a data warehouse, you will need to load historical data to your data flow as well.
Is there a calendar to choose your date range from? Refer to this article.
1. Load Historical Data to a Data Source
In some cases, it will not be possible to select your date range from a calendar like in the previous screenshot. To manually load historical data, you will need to use a date range in the source URL.
-
Go to your source, click on three dots, and select Edit. Navigate to the Advanced Settings tab to get the URL field. In the URL, the date range can be indicated by the following:
- "startDate" and "endDate"
- "Since_send_time" and "before_send_time"
- "since" and "until"
- "start" and "end"
There are two ways to define the date range, static and dynamic.
Static date range will allow you to extract data from a specific time period, in this example, data from September 1, 2020 until September 14, 2020 (written in the ISO 8601 time format).
https://us16.api.mailchimp.com/3.0/campaigns?offset=0&count=500&since_send_time=2020-09-01T00:00:00+00:00&before_send_time=2020-09-14T23:59:59+00:00
Dynamic date range extracts data relative to today’s date. In this example, we want to extract last 90 days' data, including today:
https://us16.api.mailchimp.com/3.0/campaigns?offset=0&count=500&since_send_time={{90d0}}
-
Once you define your date range, click on Save and proceed to loading your historical data via Manual Data Load button as in 1a.
Choose your snapshot keeping policy depending on your destination. Append if your destination is a dashboarding app, and Replace if your destination is a data warehouse. Confirm and click on Load data.
-
[Optional] You can check if your data extraction was successful through Show Logs and Test Extraction.
-
If your destination is a data warehouse, load historical data to your data flow (see section 2) before proceeding to the next step.
-
Return the URL back to its original state, otherwise in case of static date range, Dataddo will keep extracting data from this specific time period. In the case of dynamic date range, not changing the URL back will result in duplicities.
If you chose a dynamic date range, you don’t necessarily have to change the URL back to the original. Just keep in mind that you snapshotting policy should match your dynamic date range, e.g. weekly data extraction of the last 7 days' data.
If they don’t match, depending on your snapshotting policy, this can mean that you will be extracting the last e.g. 90 days' worth of data every day.
2. Load Historical Data to a Data Flow
This section is applicable if your destination is a data warehouse. This step is very quick as it’s simply sending the data you loaded to your source to your warehouse.
Under Flows, find the one with the source you manually loaded historical data to and click on the Manual Data Load icon.
Confirm the data insert and click on Insert data. You can check in the logs or directly in your warehouse to make sure the operation was executed successfully.
Tip & Tricks for Specific Connectors
Facebook Posts
Facebook Posts data cannot be broken down by days, you can only extract the lifetime values of your metrics such as impressions or clicks. Set the snapshot keeping policy to Append to build a time series of your data.
Google Search Console
The Google Search Console API allows loading data from the past 16 months.
Google My Business
The Google My Business API allows loading data from the past 14 months.
Instagram Story
Due to the Instagram Story API limitations, Dataddo can only access data while the story is active (24-hour window). You can create a source only when you have a story currently posted to the account.
Hourly syncing frequency is recommended in order to get data as often as possible and to get the values at the end of the 24-hour window.
If you don't expect to post a story every 24 hours, in other words we need for the source to expect empty data, enable the "allow empty" option. Cick on the three dots on your source, select Edit and scroll down on the Basic Info.
By selecting Timestamp, the data will show the metrics over time of a single story, along with the timestamp.
HubSpot
The HubSpot API does not allow selecting a date range. Dataddo will pull the full list of the selected contacts or deals.
To keep track of historical records, choose a timestamp and set the snapshot keeping policy to Append to build a time series.
TIP
To get owners' names for deals, create a new source with a specific Deal Owners dataset. Use the Data Blending feature when creating a flow, with the "owner ID" as the Join Key field.
Standard Connectors
For most of the standard connectors, you can change the date range in the URL under Advanced Settings. For Sendinblue, Simplicate, Klaviyo, and Google Search Console, dynamic data range has been implemented.
Feel free to contact us and we will help you with the setup of your sources and/or flows. To speed up the process of resolving your ticket, make sure you provide us with sufficient information.