--- title: "Transformation Pipeline" slug: "transformation-pipeline" description: "Transform your data with Dataddo easily. Our Universal Connectors allow you to get data from various sources & adjust the transformation of your existing ones." tags: ["Resource", "Universal connector", "Data transformation", "Data source"] updated: 2023-12-12T09:40:03Z published: 2023-12-12T09:40:03Z canonical: "docs.dataddo.com/transformation-pipeline" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.dataddo.com/llms.txt > Use this file to discover all available pages before exploring further. # Transformation Pipeline A **transformation pipeline** serves as a structured framework for data transformation, modeled after the concept of data processing pipelines. This structure allows API responses from third-party services to undergo multiple transformation stages, rendering the data compatible for ingestion by Dataddo. When using {{glossary.universal connector}}s, data can be fetched from diverse sources and transformed accordingly. :::(Warning) (In this section you can find:) **Guides** [How to Write a Transformation Pipeline Script](/docs/transformation-pipeline-guide): A step-by-step guide on how to write a transformation script. [Transformation Pipeline Practical Examples](/docs/transformation-pipeline-examples): A number of transformation script examples that you can try out in the Dataddo. **References** [Transformation Pipeline Stages](/docs/transformation-pipeline-stages): A comprehensive list of transformation pipeline stages and the descriptions of their functions. [Transformation Pipeline Expression](/docs/transformation-pipeline-expressions): A comprehensive list of transformation pipeline expressions and the descriptions of their functions. ::: If you are not sure where to start, when creating your ***{{glossary.data source}}***, you can click on the **Get transformation** button. In most cases, it will auto-generate a transformation script for you. ![Universal%20connector%20-%20get%20transformation](https://cdn.document360.io/084ed225-3f99-4644-a2da-39ca0cd5ef45/Images/Documentation/Universal%20connector%20-%20get%20transformation.png){height="" width=""} ## Transformation Pipeline Stages The Dataddo transformation pipeline consists of **multiple stages**. Each stage refines the API response as it progresses through the pipeline. Some stages can be used multiple times within a single pipeline. For a detailed overview of each stage, refer to [the list of transformation pipeline stages](/docs/transformation-pipeline-stages){target=`_blank`}. ## Types of Transformation Pipeline Expressions **Pipeline expressions**, employed within certain stages, define specific transformations to the input data. Expressions perform a computation or an operation on the input, and then return a value. They allow for in-memory transformations but can't access data from other objects within the pipeline. Expressions can be simple, directly referencing a value, or complex, performing operations on data. Here's a breakdown of the different types of expressions: ### Field Paths **Field paths expressions** directly reference fields within the documents that are being processed by the aggregation pipeline. They are indicated by the `$` symbol followed by the field name (e.g. `$user.name`). ### Aggregation Variables **Aggregation variables** are predefined variables which can be used in expressions. These include `$$NOW` (the current datetime), `$$ROOT` (the root document), and others. These variables provide useful shortcuts and context-aware data that can be incorporated into aggregation operations. For an in-depth explanation of these expressions and their functions, refer to [the list of operator expression](/docs/transformation-pipeline-expressions#aggregation-variables){target=`_blank`}. ### Literals **Literals** are constant values that can be used in expressions. Literals include values like strings, numbers, booleans, etc. The `$literal` operator is used when you need to include a value that starts with a `$` or a numeric/boolean value in an expression object, ensuring that it is treated as a constant rather than a field path or projection flag. ### Expression Objects **Expression objects** define expressions using field-value pairs. The values can be any valid expression, including literals, field path expressions, and operator expressions. If the value is a numeric or boolean literal, MongoDB treats it as a projection flag in the `$project` stage, unless it is wrapped in a `$literal` expression. ```json { : , ... } ``` ### Operator Expressions **Operator expressions** are similar to functions in other programming languages. They are invoked with the `$` symbol followed by the operator name and take an array of arguments. They perform various operations, such as mathematical calculations, string manipulation, logical operations, etc., and return a result. For example, `{ $add: [ "$field1", "$field2" ] }` would add the values of `field1` and `field2` together. ``` { : [ , ... ] } ``` For an in-depth explanation of these expressions and their functions, refer to [the list of operator expression](/docs/transformation-pipeline-expressions#operator-expressions){target=`_blank`}.