Data Streaming to Event Systems

By the end of this article, you will have Dataddo pushing integrated data into your event backbone, Apache Kafka or Azure Event Hubs, so any number of downstream consumers (stream processors, microservices, alerting, analytics) can subscribe to it instead of polling a database. Choose this pattern when consumers need data as a stream of messages. If the target is a warehouse table, see Batch Ingestion to Data Warehouses; for operational syncs back into business apps, see Reverse ETL (Data Activation); if an AI pipeline is the consumer, see Data Delivery to AI & Agents.

Architecture

This use case builds on the Streaming Data Flow, an existing flow type distinct from the default batch flow:

• Transfer of larger data volumes is usually considerably faster.
• No data is cached during the process, an absolute guarantee that suits compliance-sensitive pipelines.
• Because nothing is cached, features that require caching, notably flow-level data transformations and quality rules that depend on cached data, are unavailable. Extraction-level transformations and metadata columns such as the Dataddo Extraction Timestamp and Dataddo Hash still apply, because they happen at the source before the flow runs.

Data is extracted from the source on its configured schedule (see Batch Extraction) and the flow publishes it as messages to your Kafka topic or Event Hub, where your consumers take over. There is no SmartCache or staging layer in this path.

A natural pairing is a log-based CDC replication source: the destination receives the ordered list of change operations, each carrying operation-type and sequence metadata, so downstream consumers can process database changes as an event stream or materialize them on their side.

Decision Guide

Event systems are append-only logs, so there is no write-mode decision in the warehouse sense: every delivery publishes messages, and deduplication or compaction is handled on the consumer side (e.g. by keying messages consistently). The decisions that matter here are the partition key (ordering and consumer parallelism) and the message format your consumers expect.

Setup Walkthrough

1. Create a data source and set the extraction frequency to match how fresh your consumers need the stream to be (intervals as short as 1 minute are supported where the source service allows).
2. Connect the destination: your Kafka cluster or Azure Event Hubs namespace. **
3. Create a data flow and configure it as a Streaming Data Flow.
4. Configure the destination-specific options:
   • Message format for the published records.
   • Partition key, which controls how records are distributed across partitions and therefore per-key ordering for consumers.
   • Batch size (Azure Event Hubs), which tunes how many records are grouped per publish.
5. Point your consumers at the topic or Event Hub.

Configuration Recommendations