Setting Date Range

When setting up a time-sensitive data source, you have the option to define the date range for data retrieval. With each data extraction, data within this specified timeframe will be consistently loaded.

Use Case Examples

1. Daily Data Loads: Set the date range to Yesterday and the snapshotting frequency to Daily.
2. Monthly Data Loads: Use Last Month as the date range and set the snapshotting frequency to Monthly.
3. Specific Use Case: To extract weekly data, set the date range to Last 7 days and the snapshotting frequency to Weekly when creating a source. This will capture a snapshot of the past week's data every week.

You can load data from the last couple of days, last month, last year etc. This is especially useful when e.g. loading historical data

Dynamic Date Range

If the date range you need is not available in the date picker selection, you can specify a different date range in the source URL (in the Sources page, click on your data source and navigate to the Advanced Settings tab).

In these cases, your date-related parameters can be defined as part of the API call using a dynamic date range, which extracts data relative to today’s date.

The URL of a source can look like this:

https://api.service.com/endpoint?startDate={{1m1}}&endDate={{1m1}}

The expression enclosed in double curly brackets {{1m1}} consists of three parts:

1. 1m1 (the number at the start of expression): This represents the number of units (here months) to go back from the current date and defines the start of the range. 0 represents the current value.
2. 1m1 (unit definition): d = day (1d0), m = month (1m1), w = week (1w1)
3. 1m1 (the last number): Sets the end of the range. 0 marks today, 1 marks yesterday, and so on. There is a rule that the first digit has to always be larger than the second one.

Examples
API Call with 'yesterday' date range:

https://api.service.com/endpoint?startDate={{1d1}}&endDate={{1d1}}

API Call with 'this month' date range:

https://api.service.com/endpoint?startDate={{0m0}}&endDate={{0m0}}

Date String Formatting

Date string formatting is possible when APIs don't allow the standard ISO 8601 date format (YYYY-MM-DD format). In these cases, please use the following expressions:

Static Date Range

Custom date range is also used for manual data load when a calendar date range is not available. In this case, to load historical data, you will need to specify your date range in the data source URL.

Use a static date range expression when you need to extract data from a specific time period.

https://api.service.com/endpoint?startDate=2020-01-01&endDate=2020-12-31

Don't forget to change this configuration back to dynamic date range, otherwise you will keep extracting data solely from January 1, 2020 until December 31, 2020.

Related Articles

For more information on manual data load and data backfilling, see the following articles:

• Data Backfilling to Dashboarding Apps
• Data Backfilling to Storages
• Data Backfilling for Database Replication