Contents x
    Amazon S3 as a Source
    • 2 Minutes to read
    • Dark
      Light
    Contents

    Amazon S3 as a Source

    • 2 Minutes to read
    • Dark
      Light
    Article Summary

    Amazon S3 (Amazon Simple Storage Service) is a scalable cloud storage service provided by Amazon Web Services. It allows businesses and developers to store and retrieve large amounts of data in a secure and highly available manner, providing durable object storage for a wide range of use cases such as backup and restore, data archiving, content distribution, and data analytics.

    Authorize Connection to Amazon S3

    In order to connect Amazon S3 as a source, you will need to first authorize the connection to your Amazon S3.

    How to Create an Amazon S3 Data Source

    1. Under the Sources tab, click on the Create Source button and select the connector from the list.
    2. From the drop-down menu, choose your account.
      Didn't find your account?

      Click on Add new Account at the bottom of the drop-down and follow the on-screen prompts. You can also go to the Authorizers tab and click on Add New Service.

    3. Name your data source and fill in the Path of the file (you can find this information under file properties).
    4. Under File Setup (make sure you know where your file is stored):
      1. Enter your data file's name
      2. Select the appropriate file format
    5. [Optional] Configure your data transformation. If you are unsure about how to proceed, we recommend skipping this step.
    6. Configure your snapshotting preferences. Choose your sync frequency or the exact synchronization time under Show advanced settings.
    7. Preview your data by clicking on the Test Data button in the top right corner. You can adjust the date range for a more specific time frame.
    8. Click on Save and congratulations, your new data source is ready!

    Troubleshooting

    Context Deadline Exceeded Error

    ERROR CODE

    rpc error: code = DeadlineExceeded desc = context deadline exceeded

    This issue may be caused by extracting data over an extended timeframe. Use WHERE or LIMIT clauses in your SQL query to manage the size and scope of the data extraction.

    1. Use WHERE to specify the date range.
      SELECT * FROM your_table
WHERE date_column BETWEEN '202X-01-01' AND '202X-01-31';
    2. Use LIMIT to specify the specify the maximum number of records to return.
      SELECT * FROM your_table
LIMIT 1000;
    3. Combine WHERE and LIMIT for more precise control. The query in this example will return the first 1000 records where the date is after January 1, 202X.
      SELECT * FROM your_table
WHERE date_column > '202X-01-01'
LIMIT 1000;

    Related Articles

    Now that you have successfully created a data source, see how you can connect your data to a dashboarding app or a data storage.

    Sending Data to Dashboarding Apps

    Sending Data to Data Storages

    Other Resources

    Was this article helpful?
    What's Next