CSV Universal Connector

CSV, or Comma-Separated Values, is a plain-text file format used to store tabular data, where each line of the file represents a row of data, and the values in each row are separated by commas or other delimiters, like semicolons or tabs. CSV files are commonly used for data import/export, spreadsheet applications, and database management due to their simplicity and compatibility with various software systems.

How to Create a CSV Data Source

A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values and each line of the file is a data record. This guide will be divided into the following steps:

1. Fill in Basic File Information
2. Optional Steps: Define Your CSV File
3. Specify Your HTTP Request
4. Preview Data and Save Your Data Source

If you need to authorize your connection, there are several possible ways to do so. Please refer to the section on authorization methods.

Fill in Basic File Information

1. On the Sources page, click on the Create Source button and select the connector from the list.
2. Select the HTTP method to define a set of request methods.
3. Insert the URL of your CSV file. You can embed the service API authorization key or token directly in the call’s URL. For more information, refer to the section on authorization.
4. If authorization is needed, select your account.
5. Select your snapshot keeping policy.

Optional Steps: Define Your CSV File

1. Select your delimiter if it is not the default comma.
2. Select how the start of a new line is defined. This can be defined in two ways:
   1. Windows-based systems use \r\n
   2. Unix-based systems (including Linux and macOS) use \n
3. If you need to divide the file into multiple consequent queries or when you need only a part of the file
   1. Set the file offset.
   2. Specify the row limit. (For more information, see the section on how to use offset and limit.)

Specify Your HTTP Request

1. In the Transformation Pipeline window, define a short script to process the data from these APIs. You can also click on the Get transformation button. In most cases, it will auto-generate a transformation script for you.
   
2. Define your HTTP headers, name-value pairs that will be included in the request and response messages of the HTTP protocol. In most cases, you will need to set this parameter for authorization or setting the content type.
   DATADDO TIP

   For more information, refer to the section on authorization or our article on HTTP headers.

3. [Optional] On the HTTP Body tab, you can see the data transmitted in an HTTP transaction message.
4. On the Schedule tab, choose your sync frequency or the exact synchronization time.

Preview Data and Save Your Data Source

1. Make sure your transformation script has array symbols (square brackets [ ] ).
2. Click on the Get Data button to test your transformation and see a preview table.
3. Click on Save Source and congratulations, your new data source is ready!

Authorization Methods

https://api.service.com?token=YOUR_TOKEN

Authorization: Basic base64EncodedUsernameAndPassword

Authorization: Basic dXNlcjEyMzpwQHNzMTIzLg==

Authorization: Bearer YOUR_TOKEN

Troubleshooting

How to Use Offset and Limit

Offset and limit parameters are useful when you need to divide the file into multiple queries or when you need only a part of the file.

For example, due to a big data volume, you want to divide the data extraction into four batches. To do so, you will need to create four data sources.

1. Data source 1: Offset = 0, Limit = 10 000
2. Data source 2: Offset = 10 000, Limit = 10 000
3. Data source 3: Offset = 20 000, Limit = 10 000
4. Data source 4: Offset = 30 000, Limit = 10 000