Universal Connectors

Dataddo is very flexible when it comes to connecting to various online data sources. Thanks to our unique and robust data transformation middleware, Dataddo can connect to almost any online service providing it JSON/CSV/XML-based API (which is indeed the case of the overwhelming majority). Even in the case of proprietary APIs, our team is available to build custom connectors in 2-3 weeks.

Types of Universal Connectors

Each connector has some specific configuration details. For more details, see their dedicated documentation pages.

• JSON Universal Connector
• XML Universal Connector
• CSV Universal Connector

Low-level Architecture

For additional flexibility, our universal connectors are designed to mimic HTTP calls. As such, parameters such as URLs, headers, or bodies can be specified and you can connect to any service with a JSON, CSV, or XML output option.

In addition to configuring standard HTTP call parameters, it is necessary to enter a transformation pipeline expression in the form of a short script. This will indicate how to transform the output of the API call to a format that is digestible by Dataddo.

You can also use universal connectors to perform low-level API calls to services that are already covered with off-the-shelf connectors in case you need greater flexibility and customization.

Setting Up a Universal Connector

1. Sign in to your Dataddo account. Navigate to Sources and click on Create Source in the top right corner. From the list of connectors, choose one of the universal connectors.
   

2. Select the HTTP method which defines a set of request methods to indicate the desired action to be performed for a given resource.

   DATADDO TIP

   For most APIs, GET or POST will suffice. In most cases, GET should be used. For more information, please refer to the dedicated documentation page as each service has its own specifics.

   

3. Enter the URL of the API endpoint.

4. Select your account. If you haven't authorized your account yet, navigate to the Authorizers tab and click on Authorize new Service in the top right corner.

   DATADDO TIP

   For more details on authorization, please see the authorization section.

5. Choose your Snapshot Keeping Policy. You can choose from:

   • Replace (keep only the last snapshot) - only the last updated data will be stored.
   • Append (retain and merge all the snapshots) - can be used to build your own time series.

6. Name your source to label it on the top of the screen. This can be changed later.

7. (Optional step) You can choose to clone the configuration of an existing source and further transform it.
   

8. Define a short script (Transformation Pipeline) to instruct our transformation middleware to convert the API response to a format that is digestible.

9. Next, define your HTTP headers. These are the name or value pairs that are displayed in the request and response messages of message headers for HTTP Protocol. In most cases, you will need to set this parameter for authorization or setting the content type.

   Example

   if you want to send a JSON string in the body of an API call, be sure to set the content type as follows:

   Content-Type: application/json
   

10. In the HTTP Body tab, you can see the data transmitted in an HTTP transaction message. In most of the RESTful APIs, this is either a JSON or XML string. When HTTP Body is specified, the typical value for the HTTP method is either POST or PUT.

11. Continue to the Schedule tab. Choose how often and when exactly (hour, minute, timezone) your data should be synchronized.

12. You can preview your data by by entering the array (square brackets [ ] ) symbols in the Transformation Pipeline tab.

13. Click on the Get Data button to test your transformation and see a preview table.

    DATADDO TIP

    If you need to start over, click on the trash can icon in the top right corner to clear the form.

14. Finally, click on Save Source to save your selection and congratulations, your new data source is ready!

Troubleshooting

Specific Date Range Needed

If you need to specify the date range in the parameters in the URL or body, you can either use

• A static date range to get data from a specific time period
• Or a dynamic date range to generate, for example, "yesterday’s" or "last week’s" dates with every extraction

For more information, see Setting Date Range.