Contents x
Data Duplicity
  • 1 Minute to read
  • Dark
    Light
Contents

Data Duplicity

  • 1 Minute to read
  • Dark
    Light

Check Records in Your Source

If one record has multiple values from fields, that are usually for single value, such as two email adresses, phone numbers or orders, the record will be recorded twice in the table but the second one will contain the second values. This will appear as a duplicity.

Check Destination's Writemode

If the write mode for your destination is INSERT, it may cause duplicities in the table. For example, you can select UPSERT write mode instead.

Incorrect Unique Columns for UPSERT

When creating a flow and configuring the destination, for UPSERT write mode, make sure to select the correct Unique Columns. Otherwise, instead of updating the existing records and inserting the new ones, the records will be inserted again, as the unique column was not found.

Example

If you chose Customer ID as a unique column, the system will first check if the record already exists and updates in case of a change. However, if you choose Login time as the unique column, this value will not be the same and the record will be then duplicated. Read more about write modes.

Check Your Snapshotting Policy

Some combinations of snapshotting settings can lead to duplicities. Make sure you chose the right combination od Date range and Sync frequency.

Example

If you select Last 7 days for the Date Range, and Daily for Sync Frequency, you will have data of last seven days inserted each day, meaning you will have 7 times more data than you want.

Need assistance?

Feel free to contact us and we will help you with the setup. To speed the process of resolving your ticket, make sure you provide us with sufficient information.

Was this article helpful?
What's Next