Dataddo is very flexible when it comes to connecting to various online data sources. Thanks to our unique and robust data transformation middleware, Dataddo can connect to almost any online service providing it JSON/CSV/XML based API (which is indeed the case of the overwhelming majority). Even in the case of proprietary APIs, our team is available to build custom connectors in 2-3 weeks.
Types of Universal Connectors
For additional flexibility, our universal connectors are designed to mimic HTTP calls; therefore parameters such as URLs, headers, or bodies can be specified. You can connect to any service with a JSON, CSV, or XML output option.
In addition to configuring standard HTTP call parameters, it is necessary to enter a transformation pipeline expression in the form of a short script that will indicate how to transform the output of the API call to a format that is digestible by Dataddo.
You can also use universal connectors to perform low-level API calls to services that are already covered with off-the-shelf connectors in case you need greater flexibility and customization.
Setting up a Universal Connector
To create the new source, you click on New Source as usual and select one of the universal connectors.
1. HTTP Method
HTTP defines a set of request methods to indicate the desired action to be performed for a given resource. For most APIs, GET or POST will suffice. Please refer to the documentation for the specific service you want to connect to for instructions on how to configure this process correctly. However, in most cases, you need to use GET.
2. API URL
URL of the API endpoint.
Dataddo provides built-in authorization mechanisms to many online platforms and services. See the Authorization section for further detail. If you already have the account authorized, choose it from the list of authorized accounts.
4. Snapshotting Configuration
- Replace (keep only the last snapshot) - only the last updated data will be stored.
- Append (retain and merge all the snapshots) - can be used to build your own time series.
Name your source.
6. Clone Source (optional)
Optionally, you can choose to clone the configuration of an existing source and further transform it. If you do not wish to clone a source, skip this field.
7. Transformation pipeline
Since each API has a different output structure, it is necessary to define a short script that will instruct the Dataddo transformation middleware to convert the API response to a format that is digestible by Dataddo. For more information, see our Transformation Pipeline section.
8. HTTP Headers
HTTP headers are the name or value pairs that are displayed in the request and response messages of message headers for HTTP Protocol. In most cases, you will need to set this parameter for authorization or setting the content type.
For example, if you want to send a JSON string in the body of an API call, be sure to set the content type as follows:
9. HTTP Body
HTTP Body Data is the data transmitted in an HTTP transaction message. In most of the RESTful APIs, this is either a JSON or XML string. When HTTP Body is specified, the typical value for the HTTP method is either POST or PUT.
Set up the synchronization time and period, when your data will be pulled. For example, daily at 6 am, London timezone.
Once you have the URL of the API entered and all of the necessary fields filled in, you can preview a sample of the data by entering the array (square brackets [ ] ) symbols in the Transformation Pipeline section.
12. Split View
Check the box Split view to see Request and Response Previews next to each other. Otherwise, one of the request windows will be displayed in the full width of your screen.
- Request Preview - shows the request side.
- Response Preview - shows the response of the request.
13. Get Data
You can test your transformation and see a preview table of it by clicking on Get Data.
If you need to start over, select Reset form (click on the arrow next to the blue button in the top right corner).
14. Create Source
Click on Create a Source (click on the arrow next to the blue Get Data button in the top right corner) to save your selection.
If you need to specify the date period in the parameters in the URL or body, you can either specify a static date range or a dynamic date range to dynamically generate for example yesterday’s or last week’s dates on each extraction.