Custom-Schema Connectors

Custom-schema connectors in Dataddo allow you to extract data from a wide range of sources, even if they do not conform to a pre-defined schema or dataset structure. With custom-schema connectors, you have the flexibility to capture and use any data that is important to your business, regardless of its structure or format. Examples of such connectors are Google Analytics 4, Google Ads, Facebook Post, Hubspot, Netsuite, Salesforce or any database connectors (e.g. MySQL).

Prerequisites

• You have the link to API endpoint documentation.
• You have authorized a service are going to connect to.

Implementation

Flexible-schema connectors require the following workflow:

1. Call the /connectors/{connectorId}/actions/authorization endpoint for obtain the list of active authorizers. If none is available create a new one. ID of the authorizer is passed as oAuthId parameter to the payload.
2. Check the documentation for required parameters. Each of the flexible-schema connectors has a different payload structure in order to provide the desired schema flexibility. The final payload is a merge between common and connector-specific parameters.
3. Send the full payload to the /connectors/{connectorId}/preview endpoint to test the functionality of the payload. If successful, you can create the source by calling /connectors/{connectorId}/create-source endpoint.

Payload Schema

Use the schema when calling /preview and /create-source endpoints for testing the payload and creating the source. Although part of the payload is common for all flexible-schema connectors (see the code excerpt below), each custom-schema connector requires additional specific set of params to be supplied in the payload.

Common Parameters

In Dataddo, common parameters refer to a set of configuration options that are shared across all flexible-schema connectors. These parameters can be used to define common settings for each connector, such as authentication credentials or scheduling options.

POST /connectors/{connectorId}/create-source

{
   "connectorId":"CONNECTOR_ID",
   "templateId":"index",
   "label":"YOUR_LABEL",
   "type":"CONNECTOR_ID",
   "oAuthId":OAUTH_ID,
   ...
   "syncFrequency":"custom",
   "hour":"3",
   "minute":"25",
   "dayOfWeek":"*",
   "dayOfMonth":"*",
   "month":"*",
   "timezone":"UTC",
   "dateRange":"{{30d1}}",
   "strategy":"clean"
}

Connector-specific parameters

Connector-specific parameters are unique to each flexible-schema connector and are used to customize the connector's behavior and define its specific configuration options. See the examples of Snapchat, Linkedin Ads or Facebook Ads below.

Snapchat

Follow the endpoint documentation. Parameters organizationId, accountId, breakdown, metrics, labels, action_report_time, swipe_up_attribution_window and view_attribution_window are required. See full example with data types and sample values in the snippet below.

POST /connectors/snapchat/create-source
{
   ...
   "organizationId":"ORGANIZATION_ID",
   "accountId":"ACCOUNT_ID",
   "breakdown":"campaign",
   "metrics":["spend","impressions"],
   "labels":["stats_id","stats_type","stats_start_time","stats_end_time"],
   "action_report_time":"conversion",
   "swipe_up_attribution_window":"28_DAY",
   "view_attribution_window":"1_DAY"
   ...
}

Linkedin Ads

Follow the endpoint documentation. Parameters accountId, pivot, timeGranularity, metric and dataLabel are required. See full example with data types and sample values in the snippet below.

POST /connectors/linkedin/create-source
{
   ...
   "accountId":123456,
   "pivot":"campaign",
   "timeGranularity":"DAILY",
   "metric": ["comments","shares","follows","companyPageClicks","totalEngagements","costInLocalCurrency"],
    "dataLabel": ["date_start","date_stop","account_id","account_name","campaign_id","campaign_name"]
   ...
}

Facebook Ads

Follow the endpoint documentation. Parameters accountId, level, dataLabel, metric and timeBreakup required. See full example with data types and sample values in the snippet below.

POST /connectors/facebook_ads/create-source
{
   ...
   "accountId": "act_000000000",
   "level": "ad",
   "dataLabel": ["account_currency","account_id","account_name","ad_id","ad_name","adset_id","insert_date"],
   "metric": ["action_values.add_to_cart","actions.purchase","clicks","cost_per_unique_inline_link_click","cpc","ctr","spend","video_30_sec_watched_actions.video_view","video_p95_watched_actions.video_view"],
   "timeBreakup": "1",
   "breakdown": "",
   "attributionWindow": "\"28d_click\"",
   "reportTime": "conversion",
   "filter": "[{\"field\":\"publisher_platform\",\"operator\":\"CONTAIN\",\"value\":\"facebook\"}]",
   ...
}

Example

The example below demostrates the merging of common and connector-specific parameters to form a payload to be send to /connectors/{connectorId}/create-source endpoint to create a source.

POST /connectors/snapchat/create-source

{
   "connectorId":"snapchat",
   "templateId":"index",
   "label":"Snapchat Insights",
   "type":"snapchat",
   "oAuthId":OAUTH_ID,
   "organizationId":"ORGANIZATION_ID",
   "accountId":"ACCOUNT_ID",
   "breakdown":"campaign",
   "metrics":["spend","impressions"],
   "labels":["stats_id","stats_type","stats_start_time","stats_end_time"],
   "action_report_time":"conversion",
   "swipe_up_attribution_window":"28_DAY",
   "view_attribution_window":"1_DAY",
   "syncFrequency":"custom",
   "hour":"3",
   "minute":"25",
   "dayOfWeek":"*",
   "dayOfMonth":"*",
   "month":"*",
   "timezone":"UTC",
   "dateRange":"{{30d1}}",
   "strategy":"clean"
}