At the end of this guide you will have an automated pipeline that delivers data from any connector directly to a dashboarding app like Looker Studio, Power BI, Tableau, or Databox - with no data warehouse or database to provision. Choose this pattern when your datasets are small and your transformations are simple.
- If your data volume or transformation needs outgrow this pattern, ingest into a warehouse first: see Batch Ingestion to Data Warehouses.
- If you need raw data landed as files for downstream processing, see Data Lake Ingestion.
Architecture
The pipeline is source > flow > SmartCache > dashboarding app. Dataddo extracts data from your source on a schedule, stores the snapshots in SmartCache - an embedded storage system that replaces the intermediate warehouse - and your dashboarding app reads from SmartCache through the flow. This is a batch path: each scheduled extraction produces a snapshot, and the snapshotting policy decides how snapshots accumulate.
Key properties of SmartCache:
- Data residency: data is stored on Dataddo servers, and you choose from 16 global storage locations in Account Details to keep data within a specific jurisdiction.
- Security: data is encrypted at rest with AES-256 (optionally with your own HSM-backed keys) and in transit with TLS.
- Storage modes: Append (each sync adds new data to the existing set) and Replace (each sync overwrites the existing set). This is also called the snapshot keeping policy.
When SmartCache is not the right choice
SmartCache is not a replacement for a data warehouse. Move to intermediate data storage when:
- Data volume exceeds approximately 150,000 rows per source. Beyond this, dashboarding app performance degrades.
- Transformation complexity goes beyond simple joins and unions - advanced aggregations, complex join operations, or window functions belong in a warehouse.
Within those limits, Dataddo transforms data at three levels: extractions are automatically flattened and harmonized (with optional metadata columns), flows can union or join sources via Combine Data, and transformation pipeline scripts handle custom logic.
Decision Guide
Your snapshotting choice comes down to how your data behaves: use Replace when you only need the latest state (mutable data without history, or overlapping extractions), Append when each extraction brings unique new records you want to accumulate, and Append with timestamping when data changes over time and you need to track those changes. To work through the underlying questions - mutability, slowly changing dimensions, deletions, and overlapping extraction periods - see Choosing a Write Strategy.
Setup Walkthrough
- Create a data source from any connector. If you plan to track changes over time, enable the Dataddo extraction timestamp attribute during source configuration; the Dataddo Hash is also available as a generated natural key hashed from selected columns - both are described in Metadata Inclusion.
- Set the source snapshot keeping policy (Append or Replace) according to the decision guide above, and configure the date range so extractions do not overlap unintentionally.
- Connect your dashboarding app as a destination.
- Create a data flow from the source to the dashboarding app. To feed one dashboard from several sources, either combine data in a single flow (a join key for two sources, or a union of sources with identical schemas), or create one flow per source when schemas differ and no join key exists.
Configuration Recommendations
| Scenario | Snapshot keeping policy | Extraction timestamp | Date range |
|---|---|---|---|
| Latest state only (e.g. current inventory) | Replace | Not required | Any; overlaps are safe |
| Overlapping extractions, no history needed | Replace | Not required | Dynamic date range |
| Unique new records each run (e.g. daily sales) | Append | Optional | Non-overlapping periods |
Operations
Backfilling: to load historical data, trigger a Manual Data Load on the source with the snapshot keeping policy set to Append - see Data Backfilling to Dashboarding Apps.
Monitoring: use the Data Quality Watcher to get alerted on anomalies in your data before they surface in the dashboard.
Troubleshooting
- Duplicate rows in the dashboard: usually caused by an inappropriate snapshotting policy or overlapping extractions. If you do not need history, switch the policy to Replace; if you keep Append, adjust the dynamic date range so extraction periods do not overlap.
- Slow dashboard performance: your dataset has likely grown past the recommended per-source row limit. Move to a data warehouse and follow Batch Ingestion to Data Warehouses.
For generic issues (authorization, extraction failures, flow errors), see Troubleshooting.