By the end of this guide, you will have an automated pipeline that delivers timestamped CSV, JSON, JSONL, or Parquet files from any connected service into your data lake or lakehouse, organized with a file naming pattern your downstream tools can rely on, or that writes directly into Apache Iceberg tables through your catalog. Choose this pattern when your target is file-based storage such as Amazon S3 or Azure Blob Storage, or a lakehouse platform such as Databricks, Microsoft Fabric, or OneLake. If your target is a SQL data warehouse with table-level write modes, see Batch Ingestion to Data Warehouses. If you want data delivered straight into a BI tool, see Data to Dashboards.
Architecture
A pipeline consists of a data source, a data flow, and a file-based destination. The source extracts data on a schedule, creating a snapshot with each run; the flow then writes the snapshot to the lake as a file. Each write contains the full content of the most recent extraction, so every file is a self-contained, time-based snapshot of the dataset. Unlike warehouse ingestion, file delivery has no table-level write modes such as upsert; the state of your lake is controlled by how files are named and how your downstream tools consume them. The exception is Apache Iceberg delivery, which writes to catalog-managed tables and supports table write modes.
File Formats
For each flow you choose the output format:
- Parquet: an optimized columnar storage format, ideal for analytics.
- CSV: a widely used, simple format compatible with numerous tools; header row, delimiter, and date formatting are configurable.
- JSON: one JSON document per file, for consumers that expect structured objects.
- JSONL (JSON Lines): one JSON object per line, well suited to lake engines and event-style processing of large files; timestamp precision and empty-value handling are configurable.
Apache Iceberg Tables
Besides delivering raw files, Dataddo can write directly into Apache Iceberg tables through an Iceberg REST catalog. This works with any catalog implementing the standard REST specification as well as with AWS Glue via its Iceberg REST endpoint. You connect the destination with your catalog URL, warehouse, catalog credentials, and the S3 credentials for the underlying storage; each flow then targets an Iceberg table and uses regular table write modes, bringing warehouse-style semantics such as upsert to your lakehouse instead of file-snapshot management.
Dynamic File Naming and Partitioning
File partitioning splits large datasets into smaller partitions based on criteria like date, improving data organization and query performance. When configuring a flow to a file-based destination, define a custom file name pattern using dynamic variables such as {{0d0}}, {{1d1}}, {{dateRangeStart}}, {{dateRangeEnd}}, {{objectId}}, and {{objectLabel}}, or dynamic date range expressions like {{1d1}} (yesterday) and {{1m1}} (last month), with configurable date formatting, e.g. sales_{{1d1|Ymd}}.parquet. See Dynamic File Naming Patterns for the full variable and formatting reference. The same feature also works in the opposite direction: file-based sources can locate daily-generated input files by pattern.
Shaping Data Before It Lands
Even in an ELT-style lake architecture, you can transform data inside Dataddo before it is written to the lake. Extraction-level transformations handle flattening and harmonization automatically, at the flow level you can union sources with Combine Data, and transformation pipeline scripts cover custom logic. Use them where it fits your architecture; anything else can stay with the engines that read the files.
Data Lake or Data Warehouse?
Data lakes are versatile platforms for storing diverse data types and serve broad-spectrum analytics; delivery is file-based, and history is kept by accumulating snapshot files. Choose a warehouse instead when you need Dataddo to maintain a deduplicated table for you via write modes like upsert, or when you want to use Full Data Re-Sync, which is only available for destinations that support upsert. With a lake, deduplication, merging, and pruning of snapshots are handled by the tools that read the files. Apache Iceberg delivery sits in between: data lands in your lake storage, but Dataddo maintains the table through the catalog with write modes, so you get warehouse-style tables on lake infrastructure.
Choosing a File Strategy
Your file strategy depends on whether records change after extraction, whether you need a history of those changes, and whether deleted source rows must disappear from analytics. Instead of table write modes, you combine the Create New File or Replace if Exists write mode with a naming decision: timestamped file names preserve every snapshot for append or merge processing, while a static file name keeps only the latest state. Work through the four questions in Choosing a Write Strategy, then pick a configuration from the table below.
Technical metadata columns, enabled at source creation, make the snapshot files easier to process downstream. The Dataddo Extraction Timestamp and the extraction date-range boundary columns (dataddo_date_start, dataddo_date_stop) tell your lake engine exactly when each row was extracted and which period it covers, which simplifies partition reconciliation and incremental processing. The Dataddo Hash acts as a generated natural key that downstream engines can use for merge and deduplication logic.
Setup Walkthrough
- Create a data source. Set the extraction frequency and the date range each extraction should cover.
- Connect your data lake destination, e.g. Amazon S3, Azure Blob Storage, Databricks, Microsoft Fabric, or OneLake.
- Create a data flow. Set the write mode to Create New File or Replace if Exists, choose the format (CSV, JSON, JSONL, or Parquet), and define the file name pattern, including a timestamp if you want to keep historical snapshots. For Apache Iceberg, select the target table and a table write mode instead.
- Configure the consuming side. Set your analytical tool to append, merge, or pick the latest file according to your naming convention (see the configurations in Batch Ingestion to Data Lakes).
Configuration Recommendations
| Scenario | Write mode | File naming | Format | Downstream loading |
|---|---|---|---|---|
| Track changes over time (SCD) | Create New File or Replace if Exists | Timestamped, e.g. sales_{{1d1|Ymd}}.parquet |
Parquet or CSV | Append each snapshot file |
| Latest state only, drop deleted rows | Create New File or Replace if Exists | Static name, or timestamped with latest-file selection | Parquet or CSV | Replace with, or read only, the most recent file |
| Full history with consolidated view | Create New File or Replace if Exists | Timestamped | Parquet or CSV | Merge timestamped files sequentially |
| Date-partitioned lake layout | Create New File or Replace if Exists | Dynamic date range in the name, e.g. events_{{1d1|Y-m-d}}.parquet |
Parquet | Partition-aware reads by date |
Operations
- Backfilling historical data: trigger a manual extraction for the chosen date range and then a manual data insert in the flow, as described in Data Backfilling to Storages. Lake-specific note: when your file names use date variables, backfilled loads are written according to the flow's naming pattern, so verify that the resulting files land in the intended partitions and do not overwrite existing snapshots. Note that Full Data Re-Sync is not available for file-based destinations, since it requires a destination that supports upsert.
- Monitoring and quality: watch flow logs for failed writes, and use the Data Quality Firewall to validate data between extraction and write. It is available for flows delivering to storage destinations, including object storage such as Amazon S3, and runs in blocking mode (stop the transfer so no non-compliant file lands in the lake) or non-blocking mode (deliver anyway, log the discrepancies, and notify you).
Troubleshooting
- Files are overwritten instead of accumulating: the file name pattern lacks a timestamp or date variable, so each extraction replaces the previous file. Add a dynamic variable per Dynamic File Naming Patterns.
- Downstream tables show duplicates: the consuming tool is appending overlapping snapshot files. Either switch the consumer to latest-file or merge logic, or remove the overlap in the extraction date range.
- Wrong or missing input file matched (file-based sources): check the naming pattern and date format against the actual generated file names, e.g.
sales_data_{{1d1|Ymd}}.csv. - For generic issues, see Troubleshooting.
Related Connectors and Next Steps
Destination connectors: Amazon S3, Azure Blob Storage, Databricks, Microsoft Fabric, OneLake, SFTP Server.
Next steps: load into a SQL warehouse with Batch Ingestion to Data Warehouses, replicate databases with Real-Time CDC Database Replication, or feed AI pipelines with governed file delivery via Data Delivery to AI & Agents.