AFAS is a Dutch software company that provides business software solutions for various aspects of company management, including HR, finance, payroll, and CRM. Their software aims to help organizations streamline their processes, improve efficiency, and manage different aspects of their operations through integrated software tools.
Authorize Connection to AFAS
In AFAS
To authorize your AFAS account, you will need an AFAS token (= API token).
- In your AFAS account, create an App Connector that will allow Dataddo to securely extract your data. To do so, refer to the official AFAS documentation (Dutch only).
- After creating the connection, copy the AFAS token.
In Dataddo
- On the Authorizers page, click on Authorize New Service and select AFAS.
- Select your Environment Type and provide your Environment ID.
- Fill in the generated AFAS token.
- Rename your authorizer for easier identification and click on Save.
Data Coverage
AFAS uses a single flexible dataset. You choose the fields to extract when you create the source; AFAS may also expose custom fields from your account.
The available fields depend on your account and the selected dataset.
How Data Extraction Works
None of this connector's datasets use a date range. Every run pulls all currently available data.
Metadata Columns
When you create a source, you can add these Dataddo metadata columns to the extracted data:
- dataddo_hash - a fingerprint built from each record's key fields. It works as a natural key, so it is ideal for upserts (updating existing rows in your destination instead of creating duplicates).
- dataddo_extraction_timestamp - the date and time the row was extracted. Use it to track how records change over time, for example to build slowly changing dimensions.
How to Create an AFAS Data Source
Creating a data source takes you through five steps, shown in the progress bar at the top of the wizard. Each step is explained below.
1. Pick the connector
On the Sources page, click Create Source, then select the connector from the catalog. Use the search bar or the category tabs if you do not see it right away. You can rename the source at any time using the pencil icon next to its name.
2. Choose the account
This step selects what Dataddo reads from.
- Authorizer: Select an account you have already authorized from the drop-down. If you have none yet, choose Add new account and follow the prompts. If no authorizer is selected, Dataddo asks you to authorize before you continue.
- What to extract from: Select the exact entity you want to pull data from. Depending on the service this may be labelled an account, property, profile, workspace, or similar, sometimes with a sub-level to choose as well.
- Multiple accounts: To pull the same data from every entity you can access, turn on Automatically collect data from all .... This is multi-account extraction. Leave it off to choose them by hand.
3. Build the data model
Choose exactly what to extract. What you can pick depends on the connector, and the wizard may group the fields differently for each one.
- Values: the measures you want, often called metrics, such as sessions, clicks, or revenue.
- Breakdowns: the fields you group or split those values by, often called dimensions or attributes, such as date, country, or campaign.
- Context metadata (when available): identifier fields that Dataddo derives from your selection, such as the account or property ID. They let you tell rows apart when you combine several sources.
Some connectors limit which fields can be queried together; the wizard flags this where it applies. Click Test on Sample Data at any point to preview the result before you continue.
4. Add metadata columns (optional)
Two optional columns help your destination handle the data.
- Dataddo Hash (Include Row Hash): a fingerprint built from the columns you pick. It works as a natural key, so your destination can deduplicate rows and run upserts instead of creating duplicates. Turn it on, then select the columns that uniquely identify a row.
- Dataddo Extraction Timestamp: the time each row was extracted. Use it to watermark the data, for example to build slowly changing dimensions or to track when a value last changed.
5. Set the schedule
Decide how often Dataddo runs the extraction.
- Frequency: how often the pipeline runs, for example daily. Click Show advanced settings to also set the exact hour and minute (UTC).
- Date range: the relative window each run extracts, for example "Yesterday". The window moves forward on every run.
- Historical data: a new source starts from the current window. To load older data, run a full data re-sync after the source is created.
- Allow Empty Data Extractions: when on, a run that returns no data records zero rows instead of failing. Turn it on if the source can legitimately have periods with no data.
Click Save. Your data source is ready.
How to Load All Your Records
To load historical data, we recommend first checking if fields such as created on or last modified on can be added to your GetConnector. These fields should be available for nearly all GetConnector definitions.
Once all your historical data is extracted, the "created on" or "last modified on fields will allow you to sort your data according to date. As a result, you can keep extracting only the most recent records.
If these fields are not available, you can sort data only according to e.g. ID. In this case, you won't be able extract the most recent data after you are done with historical data load. Proceed with the alternative approach instead.
Load All Data in Batches
Loading AFAS historial data will be done in batches.
Load your historical data to a data warehouse using the upsert write mode to make sure that Dataddo will take care of any duplicates for you.
1. Increase the limit to 50,000 records
By default, the AFAS connector will extract 10,000 records. Increase the limit to 50,000 (recommended maximum) in your source's advanced settings.
Click on your source and navigate to the Advanced Settings tab and change the limit to 50,000 records.
2. Run manual data extraction
Click on the manual data load button to extract the initial 50,000 record. You can check if your extraction was succesful in the logs.
3. Load historical data to your data warehouse
If you are using a data warehouse, navigate to your flow and click on the manual data insert button.
4. Go back to your source and increase the Skip field by 50,000
Navigate back to the Advanced Settings tab of your source and increase the Skip field by 50,000. Next extraction job will skip the previously extracted data and will extract the next 50,000 records.
5. Perform manual data load for your source and flow again
Repeat steps 2-5 until there is no more data left or until you get a number lower than the limit (50,000). Don't forget to increase the Skip field by 50,000 every time or insert your data using upsert write mode to avoid duplicates.
The total number of records of your specific GetConnector is 323,000, you will need to extract data in 7 batches. As the last batch will only produce 23,000 records, you will need to re-adjust the limit as per step 1.
In some cases, sorting data by ID or a similar field will make more sense for the initial historical data load.
Setting up Daily Increments
Once all historical data has been loaded, return your source's settings to the initial state to extract the latest records.
If you expect a few thousands records a day, we recommend setting your limit to 10,000-20,000. Otherwise, extract full 50,000 records daily, in which case we recommend using the the upsert write mode to insert data into your data warehouse.
If you want to extract only yesterday's data (including data what was modified yesterday), we recommend trying a JSON filter. To set a JSON filter up, refer to the official AFAS documentation .
This filter can be applied only when creating a new source. After loading all your historical data, set up a new source with the JSON filter and connect it to your data warehouse. Don't forget to delete the original source.
Alternative Approach
In case you cannot sort data using the created on or last modified on fields, you will need to create multiple sources for your GetConnector and extract all data every time. The sources should look as the following:
- 1st source - limit: 50,000, skip: 0
- 2nd source - limit: 50,000, skip: 50,000
- 3rd source - limit: 50,000, skip: 100,000
- Etc. (until you extract all records)
Add all of these sources created for a specific GetConnector into a single flow. The data will be unioned into a single table which will then be inserted into your data warehouse.
You will need to repeat this process for all the GetConnectors that require this approach.
Troubleshooting
Data Preview Unavailable
No data preview when you click on Test Data might be caused by an issue with your source configuration. The most common causes are:
- Date range: Try a smaller date range. You can load the rest of your data afterward using manual data load.
- Insufficient permissions: Please make sure your authorized account has at least admin-level permissions.
- Invalid metrics, attributes, or breakdowns: You may not have any data for the selected metrics, attributes, or breakdowns.
- Incompatible combination of metrics, attributes, or breakdowns: Your selected combination cannot be queried together. Please refer to the service's documentation to view a full list of metrics that can be included in the same data source.
Related Articles
Now that you have successfully created a data source, see how you can connect your data to a dashboarding app or a data storage.
Sending Data to Dashboarding Apps
Sending Data to Data Storages
Other Resources