Release Note 2024-11

We’re excited to bring you the latest updates to Dataddo! The November 2024 release continues our commitment to optimizing the Flows page, introducing new improvements designed to provide a clearer, more comprehensive understanding of your data at a glance. Alongside these enhancements, we’ve also made key adjustments to streamline your experience and addressed various bugs for improved performance.

Here’s a detailed overview of the new features, UI updates, and bug fixes included in this release.

Connectors

New Connectors:

• DynamoDB
• LocalFalcon
• Survicate
• Zenvoices

New Datasets:

• Instagram Media: Added the Comments dataset.
• LinkedIn Organic: Added the Audience Insights dataset.

Connector Updates:

• TikTok: The Dataddo Hash field is now available.
• YouTube Analytics: The Videos - Retention dataset received a new emitter and updates, with corresponding updates applied to all sources using this dataset in Dataddo.

New Features

Full Data Re-Sync

Load historical data all at once with our new full data re-sync functionality. Previously, users needed to determine the maximum data load for each connector and manually chunk data, which could be challenging. The system now automatically determines the limitations and optimally distributes the requests to sync all available data efficiently.

Conditions for Use:

• Users must have write permissions (e.g., the user can and has configured upsert write mode for some flows). The system will verify this before enabling the feature.

This functionality is being implemented gradually as each connector has different max loads and is designed to eventually replace the ad-hoc manual data load process.

Sensitive Data Handling

The advanced settings have been streamlined to enhance data security and usability. A new source set-up step will soon enable users to define how detected sensitive data is handled, including options for hashing sensitive data for added security.

Previously, advanced settings allowed only adding metadata such as Property ID, Dataddo Insert Date, and Dataddo Hash (a unique identifier or composite key for upsert).

UI Changes and Enhancements

Ongoing Enhancements to the Flows Page

We continue to enhance flow-level data movement capabilities, delivering improved observability and clearer, at-a-glance insights into the origins, destinations, and pathways of your data.

Flow Grid Write Modes:

• The flow grid now displays write modes more comprehensively, including streaming mode, in addition to upsert, truncate, and truncate-insert. This enhancement provides greater transparency and helps maintain data quality.

Destination Descriptors:

• The flows page now includes destination descriptors, allowing you to see which flow is sending data to specific data warehouses (e.g. identifying different projects, databases, and tables within GBQ) all on one page.

Comprehensive Schema Overview

Fixed-schema connectors are also being continuously improved to provide a more comprehensive view of schema details. This will allow users to better understand the data they’re working with.

Apart from the full list of available attributes, the source setup now displays:

• Data types for each field.
• Tooltips explaining each field and the type of data being extracted.
• Indicators for sensitive data, tying into the upcoming sensitive data handling feature.
• Attribute relations and dependencies (viewable by hovering over the attribute name), showing connections to other datasets to help:
  • Identify any additional required data for joins.
  • Address data gaps for a clearer understanding of how data works together.

In development:

• The enforcement of correct data types will be gradually implemented across individual connectors to ensure system stability.
• The comprehensive dataset view is being progressively rolled out to all fixed-schema connectors, expanding beyond the 50% of templates currently covered.

These updates aim to give users a clearer and more actionable understanding of their data structure and relationships.

Additional UI Updates

New Dashboard Controller:

• A new feature displays the number of broken authorizers, sources, destinations, and flows. Users can click on Debug to view a detailed list of the affected components and repair them.

Snapshotting Tab:

• Date range selection is now conveniently displayed on the source details page, making it easier to review and manage snapshotting configurations.
  

Platform Optimization

To ensure smoother performance and optimize system efficiency, only sources actively used in flows will now be regularly synced. This update allows us to focus resources on syncing the data that matters most, improving reliability and overall platform performance.

In Development

We are actively preparing additional enhancements to streamline data movement observability and management:

Enhanced Flow Type Information:

• Flow types will soon feature distinctive icons, and eventually, the write mode (append or replace) will be displayed directly at the flow level. This information will be moved from the source level to the flow level for greater clarity.
• All data movement configuration details will ultimately be consolidated on the Flows page.

Enhanced Source Configuration Insights:

• The Sources page will include what account the data is extracted from (e.g., identifying specific HubSpot accounts in multi-account setups) and the associated configuration settings. The rollout of descriptors will be gradual, with custom implementations tailored to each connector.

Stay tuned as we continue refining these functionalities to further optimize your data management experience.

Bug Fixes

Podbean, Google Analytics 4, ActOn, and LinkedIn:

• Resolved reauthorization issues where users received an error notification indicating the account could not be found. The issue was identified and fixed within 24 hours of detection.

TikTok:

• Fixed an issue where connectors showed successful extractions, but no data was retrieved. The connector was promptly updated to resolve the problem.

LinkedIn Organic:

• Added missing facets to the Audience Insight dataset for improved targeting insights:
  • urn:li:adTargetingFacet:companyCategory
  • urn:li:adTargetingFacet:jobFunctions

Google Merchant Center:

• Resolved a pagination issue by implementing a cursor-based pagination parameter (pageToken). The connector now successfully paginates results with each API call using the updated loader file.