Load Historical Data Using Manual Data Load
- 3 Minutes to read
- DarkLight
Load Historical Data Using Manual Data Load
- 3 Minutes to read
- DarkLight
Article Summary
Dataddo allows you to manually load historical data from a specific date range.
Do a one-time request in the Dataddo app and the manual data load will affect all flows and destinations connected to the source.
It is possible to do a manual data load any time after you create your source. Please keep in mind that if your destination is a data warehouse, you will need to load historical data to your data flow as well. No calendar to choose your date range from? Refer to this article.No Calendar Date Range
1. Load Historical Data to a Data Source
After you create your source, go to your list of sources and click on the Manual Data Load icon.
Set up the data load: choose the limit, date range, snapshot keeping policy, and Confirm. If your destination is a data warehouse, you will also need to load your historical data to your flow as well (see section 2).
Choose the snapshot keeping policy depending on your destination.
DATADDO TIP
In the case of a dashboarding app, Append should be used in most cases. As your snapshots are saved in the source itself, historical data will be added to your existing data/snapshots.
If your destination is a data warehouse, you should use Replace since there is no need for snapshots to be saved in the source.
2. Load Historical Data to a Data Flow
This section is applicable if your destination is a data warehouse. This step is very quick as it’s simply sending the data you loaded to your source to your warehouse.
Under Flows, find the one with the source you manually loaded historical data to and click on the Manual Data Load icon.
Confirm the data insert and click on Insert data. You can check in the logs or directly in your warehouse to make sure the operation was executed successfully.
Tip & Tricks for Specific Connectors
Facebook Pages
The Facebook Pages API requires loading historical data in batches of 90 days, and up to 5 years back.
Facebook Ads
The Facebook Ads API limits the historical data load to the last 37 months (i.e. 3 years).
Google Analytics
Due to the Google Analytics API's limitation
- Some combinations of metrics will not work (check Google Analytics official documentation for the metrics that can be queried together)
- A maximum of 10 metrics and 7 dimensions can be selected per source
You can edit the limit of how much data is being loaded for the historical load. If you see a round number (e.g. 10 000) in the logs, most likely not everything was loaded.TIP: How can I see if I've reached the limit?
Check how much daily data is being loaded when creating the source in the data preview, or in the logs. Go to Sources, click on the three dots next to your source, and select Show Logs to see the row counts from the last load.
Google Analytics 4
The Google Analytics 4 API allows loading data only from the past year.
Instagram Business (page)
The limit for Instagram Business historical data is 30 days. As a result, it's possible to load the data in batches of 30 days and go as far as one year back. "New Followers Count" Metric Exception
The Instagram Business API limits this metric to only pull the number of followers each day and can only load the last 30 days. It is not possible to load older data for this metric. Set your snapshot keeping policy to Append and build a time series of your data.
When loading Pinterest historical data, the data must be loaded in batches of 6 months. However, there is no limit to how far back you can go.
Standard Connectors
For most of the standard connectors, you can change the date range in the URL under Advanced Settings. For Sendinblue, Simplicate, Klaviyo, and Google Search Console, dynamic data range has been implemented.
For more information, head over to Manual Data Load Without Calendar Date Range.
Need assistance?
Feel free to contact us and we will help you with the setup of your sources and/or flows. To speed up the process of resolving your ticket, make sure you provide us with sufficient information.
Was this article helpful?
