Universal Connectors

Dataddo is very flexible when it comes to connecting to various online data sources. Thanks to our unique and robust data transformation middleware, Dataddo can connect to almost any online service providing it JSON/CSV/XML based API (which is indeed the case of the overwhelming majority). Even in the case of proprietary APIs, our team is available to build custom connectors in 2-3 weeks.

Types of Universal Connectors

• CSV
• XML
• JSON

Low-level Architecture

For additional flexibility, our universal connectors are designed to mimic HTTP calls; therefore parameters such as URLs, headers, or bodies can be specified. You can connect to any service with a JSON, CSV, or XML output option.

In addition to configuring standard HTTP call parameters, it is necessary to enter a transformation pipeline expression in the form of a short script that will indicate how to transform the output of the API call to a format that is digestible by Dataddo.

You can also use universal connectors to perform low-level API calls to services that are already covered with off-the-shelf connectors in case you need greater flexibility and customization.

Setting up a Universal Connector

To create the new source, you click on New Source as usual and select one of the universal connectors.

1. HTTP Method

HTTP defines a set of request methods to indicate the desired action to be performed for a given resource. For most APIs, GET or POST will suffice. Please refer to the documentation for the specific service you want to connect to for instructions on how to configure this process correctly. However, in most cases, you need to use GET.

2. API URL

URL of the API endpoint.

3. Authorization

Dataddo provides built-in authorization mechanisms to many online platforms and services. See the Authorization section for further detail. If you already have the account authorized, choose it from the list of authorized accounts.

4. Snapshotting Configuration

• Replace (keep only the last snapshot) - only the last updated data will be stored.
• Append (retain and merge all the snapshots) - can be used to build your own time series.

5. Name

Name your source.

6. Clone Source (optional)

Optionally, you can choose to clone the configuration of an existing source and further transform it. If you do not wish to clone a source, skip this field.

7. Transformation pipeline

Since each API has a different output structure, it is necessary to define a short script that will instruct the Dataddo transformation middleware to convert the API response to a format that is digestible by Dataddo. For more information, see our Transformation Pipeline section.

8. HTTP Headers

HTTP headers are the name or value pairs that are displayed in the request and response messages of message headers for HTTP Protocol. In most cases, you will need to set this parameter for authorization or setting the content type.

For example, if you want to send a JSON string in the body of an API call, be sure to set the content type as follows:

Content-Type: application/json

9. HTTP Body

HTTP Body Data is the data transmitted in an HTTP transaction message. In most of the RESTful APIs, this is either a JSON or XML string. When HTTP Body is specified, the typical value for the HTTP method is either POST or PUT.

10. Schedule

Set up the synchronization time and period, when your data will be pulled. For example, daily at 6 am, London timezone.

11. Preview

Once you have the URL of the API entered and all of the necessary fields filled in, you can preview a sample of the data by entering the array (square brackets [ ] ) symbols in the Transformation Pipeline section.

12. Split View

Check the box Split view to see Request and Response Previews next to each other. Otherwise, one of the request windows will be displayed in the full width of your screen.

• Request Preview - shows the request side.
• Response Preview - shows the response of the request.

13. Get Data

You can test your transformation and see a preview table of it by clicking on Get Data.

If you need to start over, select Reset form (click on the arrow next to the blue button in the top right corner).

14. Create Source

Click on Create a Source (click on the arrow next to the blue Get Data button in the top right corner) to save your selection.