Universal connector

 

Free consultation available!

Feel free to contact us. We are happy to help you configure the Universal connector.

 

Dataddo is extremely flexible when it comes to connecting to various online data sources. Thanks to our unique and robust data transformation middleware, Dataddo can connect to almost any online service providing it has JSON/CSV/XML based API (which is indeed a case of overwhelming majority). Even in the case of proprietary APIs, our team is available to build custom connectors in 1-2 business days.

Low-level architecture

For additional flexibility, our universal connectors are designed to mimic HTTP calls; therefore parameters such as URLs, headers or bodies can be specified. You can connect to any service with a JSON, CSV or XML output option.

In addition to configuring standard HTTP call parameters, it is necessary to enter a transformation pipeline expression in the form of a short script that will indicate how to transform the output of the API call to a format that is digestible by Dataddo.

You can also use universal connectors to perform low-level API calls to services that are already covered with off-the-shelf connectors in case you need greater flexibility and customization.

Setting up the connector

To create the new source, you click on New Source as usually and select JSON Universal.

API URL

URL of the API endpoint.

Authorization

Dataddo provides built-in authorization mechanisms to many online platforms and services. See the Authorization section for further detail.

Transformation pipeline

Since each API has a different output structure, it is necessary to define a short script that will instruct the Dataddo transformation middleware to convert the API response to a format that is digestible by Dataddo. For more information, see our Transformation Pipeline section.

HTTP Method

HTTP defines a set of request methods to indicate the desired action to be performed for a given resource. For most APIs, GET or POST will suffice. Please refer to the documentation for the specific service you want to connect to for instructions on how to configure this process correctly. In most cases, you will need to use GET.

HTTP Headers

HTTP headers are the name or value pairs that are displayed in the request and response messages of message headers for HTTP Protocol. In most cases, you will need to set this parameter for authorization or setting the content type.

For example: if you want to send a JSON string in the body of an API call, be sure to set the content type as follows:

Content-Type: application/json

HTTP Body

HTTP Body Data is the data transmitted in an HTTP transaction message. In most of the RESTful APIs, this is either a JSON or XML string. When HTTP Body is specified, the typical value for the HTTP method is either POST or PUT.

{"data":"some value"}

Preview Data

HTTP Body Data is the data transmitted in an HTTP transaction message. In most of the RESTful APIs, this is either a JSON or XML string. When HTTP Body is specified, the typical value for the HTTP method is either POST or PUT.

 

Once you have the URL of the API entered, the HTTP method is set to GET, you can preview a sample of the data by entering the array (square brackets [] ) symbols in the Transformation Pipeline section.

Screenshot 2021-06-24 at 17.29.41

Example

In this example, we will fetch landslide data from NASA, setting the Transformation Pipeline so that we only extract the metrics relevant for us.

The URL of the API endpoint is https://data.nasa.gov/resource/tfkf-kniw.json .

As these data are freely accessible, we do not need any authorization.

The HTTP Method needs to be set to GET as we want to fetch data.

The HTTP Headers and Body can be left empty. 

As for the Transformation Pipeline, let's say we want to collect country name, event date, event title, fatality count, injury count, landslide size, location description and source name. Then the script should look something like this:

[
{"$unwind":"$data"},
{"$project": {
"country_name": {"$ifNull":["$data.country_name",""]},
"event_date": {"$ifNull":["$data.event_date",""]},
"event_title": {"$ifNull":["$data.event_title",""]},
"fatality_count": {"$ifNull":["$data.fatality_count",0]},
"injury_count": {"$ifNull":["$data.injury_count",0]},
"landslide_size": {"$ifNull":["$data.landslide_size",""]},
"location_description": {"$ifNull":["$data.location_description",""]},
"source_name": {"$ifNull":["$data.source_name",""]}
}}
]

Once all these parameters are set and the source is labeled, we can click on Connect, preview the data and save them. Congratulations, you have created a data source!

 

nasa