The AWS Glue destination writes your data as tables in the AWS Glue Data Catalog. Dataddo registers each table in a Glue database and stores the data in the S3 location that backs it, so the tables are ready to query with engines such as Amazon Athena.
Supported Data Format
This destination writes Apache Iceberg tables only. Dataddo manages the Iceberg metadata and data files and registers the table in the Glue Data Catalog. There is no CSV, Parquet, or JSON file option here, because the storage layout is defined by the Iceberg table format.
Authorize Connection to AWS Glue
Prerequisites
- An AWS account with the AWS Glue Data Catalog enabled.
- An IAM user or access key with permission to create and update Glue databases and tables, and to read and write the S3 location behind them.
Create the authorizer
Go to Authorizers > Authorize New Service > AWS Glue, then fill in the fields below and save.
| Field | Required | Description |
|---|---|---|
| Key | Yes | The AWS access key ID. It cannot contain whitespace. |
| Secret | Yes | The AWS secret access key. It cannot contain whitespace. |
| Region | Yes | The AWS region of the Glue Data Catalog, for example us-east-2. It cannot contain whitespace. |
Create an AWS Glue Destination
Go to Destinations > Create Destination > AWS Glue, choose the authorizer you created, then set the fields below.
| Field | Description |
|---|---|
| Destination Name | A name for this destination in Dataddo. |
| Database | The Glue database that holds your tables. The value cannot contain whitespace. |
Write Modes
Write mode decides how each run changes the target table. The available modes are resolved by Dataddo's writer at flow time, so the exact list can depend on the current table state. The flow default is insert.
| Mode | Meaning |
|---|---|
insert |
Appends the run's rows to the table. |
truncate_insert |
Empties the table, then writes the current data. |
Table Naming
The Table field on the flow sets the table name. Dataddo writes to database.table, where the database comes from the destination and the table name comes from the flow. The table is registered in the Glue database on first write when it does not exist yet.
How to Create a Flow to AWS Glue
- Go to Flows > Create Flow.
- Add one or more sources.
- Add the AWS Glue destination.
- Choose the write mode.
- Enter the table name.
- Set the schedule and save.
Troubleshooting
- Database or table rejected. The database name cannot contain whitespace. Use a single token such as
my_database. - Access denied. Confirm the access key has permission to manage Glue databases and tables and to read and write the backing S3 location.
- Wrong region. Confirm the Region matches the region of your Glue Data Catalog.