--- title: "Amazon S3" slug: "s3" description: "Get your data from any source, including data warehouses, to S3 quickly with Dataddo! Create secure and reliable data flows or replicate databases easily. " tags: ["Data destination", "How-to guide", "Data lake"] updated: 2024-06-14T09:29:11Z published: 2024-06-14T09:29:11Z --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.dataddo.com/llms.txt > Use this file to discover all available pages before exploring further. # Amazon S3 **Amazon S3 (Simple Storage Service)** is a scalable cloud storage service provided by Amazon Web Services (AWS). It offers a way to store and retrieve data, such as files, images, videos, and backups, in a highly durable and easily accessible manner, making it a foundational component for various cloud-based applications and services. ## Prerequisites - You have created an AWS S3 bucket. - You have created [an IAM user](/docs/s3#in-aws-portal) with **AmazonS3FullAccess** policy attached to your bucket. ## Authorize Connection to Amazon S3 ### In AWS Portal #### Create S3 Bucket 1. Sign in to the **AWS Management Portal** and navigate to **S3 Console**. 2. Click on **+Create bucket**. Name your bucket and choose **AWS Region**. You will need to [provide this information to Dataddo](/docs/s3#in-dataddo). 3. Click on the **Create** button to finalize the process. #### Configure the Access Permissions 1. Navigate to the **IAM service**, click on **Users** and continue with **Add User**. 2. Name your user and for the access type, select **Programmatic access**. 3. In the **Set permissions** step, choose **Attach existing policies directly**. Attach a policy that grants the necessary S3 permissions. This could be one of the following: - **The AmazonS3FullAccess** policy, which grants full access to all S3 resources. - **A custom policy** that only grants access to the necessary bucket and actions. 4. **Save** the IAM user's credentials. You will be given an **access key** and a **secret key**. These credentials will be needed for configuring the connection to your S3 bucket in Dataddo. Use the template below for configuring the **Access Policy**. Make sure to replace **your-bucket-name** with your bucket identifier. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl", "s3:GetObject", "s3:GetObjectAcl", "s3:DeleteObject", "s3:ListBucket", ], "Resource": "arn:aws:s3:::your-bucket-name/*" } ] } ``` ### In Dataddo 1. In the **Authorizers** tab, click on [Authorize New Service](https://app.dataddo.com/service/new) and select **S3**. 2. You will be asked to fill the following fields 1. **Bucket**: Provide the identifier of S3 bucket you want to use for reading or writing the data. 2. **Region**: Region of the S3 bucket. 3. **Key**: Provide your [AWS Access Key](/docs/s3#in-aws-portal). 4. **Secret**: Provide your [AWS Secret Key](/docs/s3#in-aws-portal). 3. Click on **Save**. ## Create a New S3 Destination 1. Under the **Destinations** tab, click on the [**Create Destination**](https://app.dataddo.com/destinations) button and select the destination from the list. 2. **Select your account** from the drop-down menu. 3. Fill in the **Path**. Use the **name of the folder in your Container + slash** (e.g. "database/" ). 4. Name your destination and click on **Save** to create your destination. Need to authorize another connection? Click on **Add new Authorizer** in drop-down menu during authorizer selection and follow the on-screen prompts. You can also go to the **Authorizers** page and click on [**Add New Service**](https://app.dataddo.com/service/new). ## Creating a Flow to S3 Storage 1. Navigate to **Flows** and click on [**Create Flow**](https://app.dataddo.com/flow/new). 2. Click on **Connect Your Data** to add your ***source(s)***. 3. Click on **Connect Your Data Destination** to add the ***destination***. 4. Choose the [write mode](https://docs.dataddo.com/docs/data-storages#write-modes) and fill in the other required information. 5. Check the **Data Preview** to see if your configuration is correct. 6. **Name** your flow and click on **Create Flow** to finish the setup. ### File Partitioning File partitioning **splits large datasets into smaller, manageable partitions**, based on criteria like date. This technique enhances data organization, query performance, and management by grouping subsets of data with shared attributes. During ***flow*** creation, you can: - Select one of the **predefined file name patterns**. - Define your own **custom name** to suit your partitioning needs. **Example of a custom file name** When creating a custom file name, use variations of the offered file names. For example, use a **base file** name and add a different **date range** pattern: ``` xyz_{{1d1|Ymd}} ``` Using this file name, Dataddo will create a new file named `xyz` every day, e.g. `xyz_20xx0101`, `xyz_20xx0102` etc. For more information, see [Dynamic File Naming Patterns](/docs/dynamic-file-naming-patterns). --- ## Troubleshooting ### Error Message `kms:GenerateDataKey` **ERROR MESSAGE** ``` Action failed: stream transfer: write data from stream: upload JSON to s3: uploading data to S3 bucket 'bucket-name': operation error S3: PutObject, https response error StatusCode: 403, RequestID: request_id, HostID: host_id api error AccessDenied: User: user_name is not authorized to perform: kms:GenerateDataKey on resource: arn:aws:s3:::bucket-name/ because no identity-based policy allows the kms:GenerateDataKey action ``` This issue is most likely caused by your buckets utilizing special encryption methods. To solve this, you'll need to include the `kms:GenerateDataKey` permission scope to your IAM user using, for example, this template: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "kms:GenerateDataKey", "Resource": "arn:aws:s3:::your-bucket-name/*" } ] } ``` ## Related Articles - [Data Backfilling to Storages](https://docs.dataddo.com/docs/data-backfilling-to-storages) - [Write Modes](https://docs.dataddo.com/docs/data-storages#write-modes) - [Implementation of Batch Ingestion to Data Lakes](https://docs.dataddo.com/docs/batch-ingestion-to-data-lakes) - [Network Access Control List (ACL) Configuration](https://docs.dataddo.com/docs/network-acl) - [SSH Tunnelling](https://docs.dataddo.com/docs/ssh-tunnelling) - [Data Transformations](https://docs.dataddo.com/docs/data-transformations) - [Data Quality Firewall](https://docs.dataddo.com/docs/data-quality-firewall)