Postgres WAL

Postgres WAL (Write-Ahead Logging) is a system used by PostgreSQL to ensure data integrity by recording changes to the database before they are written to the actual data files. This logging process aids in recovering from crashes by replaying these logs to rebuild the database state up to the point of failure.

The main function of the Postgres WAL source is to replicate DELETE operations.

Authorize Connection to Postgres

To authorize the connection to your Postgres server, please follow the authorization guide on our Postgres as a destination article.

Additional User Permission

Apart from the general user permissions (CREATE, SELECT, UPDATE, INSERT, DELETE), you will need to add the following:

• Logical replication permission

  ALTER ROLE dataddo REPLICATION;
  

Prerequisite: Unique Field Requirement

To set up a Postgres WAL source , it is essential to define a unique field, preferably a PRIMARY KEY, as there is no support for implicit replica identity.

How to Create a Postgres WAL Data Source

1. Under the Sources tab, click on the Create Source button and select the connector from the list.
2. From the drop-down menu, choose your account.
   Didn't find your account?

   Click on Add new Account at the bottom of the drop-down and follow the on-screen prompts. You can also go to the Authorizers tab and click on Add New Service.

3. Name your data source.
4. Select your table and columns.
5. Configure your snapshotting preferences. Choose your sync frequency or the exact synchronization time under Show advanced settings.
6. Preview your data by clicking on the Test Data button in the top right corner. You can adjust the date range for a more specific time frame.
7. Click on Save and congratulations, your new data source is ready!

Troubleshooting

Insufficient Disk Space

The Dataddo source can be configured to connect only every hour (not frequently). For this reason, your server requires sufficient disk space to store WAL files for at least 1 to 3 hours.

The additional storage time depends on the volume of changes (INSERT, UPDATE, DELETE) occurring in your database. Make sure you estimate the disk capacity accordingly to cover any unexpected traffic peaks or large maintenance operations performed by your DBAs and other personnel.

No Special Configuration for Replica Identity

Currently, there is no support for explicit replica identity settings to use with the WAL source.