Contents x
    Azure Blob Storage
    • 6 Minutes to read
    • Dark
      Light
    Contents

    Azure Blob Storage

    • 6 Minutes to read
    • Dark
      Light
    Article Summary

    Azure Blob Storage is a cloud-based object storage service provided by Microsoft Azure. It allows users to store and manage unstructured data, such as documents, images, videos, and backups, in a highly scalable and secure manner, offering high durability, accessibility, and security and making it ideal for a wide range of applications and scenarios in the cloud.

    Prerequisites

    • You have already created a Storage Account and within that you have at least one Container.
    • You have allowed incoming connection to your server from Dataddo IPs.
    • You have generated a Shared Access Token to authorize the connection to your storage.

    Authorize Connection to Azure Blob Storage

    In Azure Portal

    Add a Storage Account

    1. In Azure Portal in the Storage Accounts section, click on Create.
    2. Fill in Storage account name. You will need to provide this value during configuration in Dataddo.
    3. Use the default configuration. In case you want to fine-tune it, make sure following configuration is set:
      1. In the Advanced tab: allow both Enable storage account key access and Allow enabling public access on individual containers.
      2. Make sure that in the Networking tab, Enable public access from all networks is selected.
    4. Review and create the Storage Account.

    Add a Container in Storage Account

    1. In Azure Portal's Storage Accounts section, select the storage account you want to use, go to Containers and Add new container
    2. Set the name of the container. You will need to provide this value during configuration in Dataddo.
    3. Use the default Private (no anonymous access) access configuration.

    Generate Shared Access Token

    1. In the Storage Account section, select Container you want to use and under Settings click on Shared access tokens.
    2. Set Permissions to Read, Add, Create, Write, Delete, List and Move.
    3. Set Expiry to value to comply with your security policies. We recommend to set it on at least 6 months (after the token expiration you will need to reauthorize the connection).
    4. Set Allowed IP addresses to empty value. We recommend this because you can only set IP address range and not list multiple IP addresses. The connection is secured via unique token so there are no security implications.
    5. Set Allowed protocols to HTTPS only.
    6. Click on Generate and make sure to copy to copy the value of Blob SAS Token. You will need to provide this value during configuration in Dataddo.

    Azure Blob Storage - Token&URL

    In Dataddo

    1. In the Authorizers tab, click on Authorize New Service and select Azure Blob Storage.
    2. You will be asked to fill the following fields
      1. Storage Account. Provide the name of the storage account you want to use.
      2. Container. Provide the name of the container you want to use for reading or writing the data.
      3. Shared Access Token. Provide the value of the shared access token.
    3. After the account configuration you will be redirected back to Destination page where you will need to fill the Path. Use name of the folder in your Container + slash (e.g. database/).
    4. Click on Save.

    Create a New Azure Blob Storage Destination

    1. On the Destinations page, click on the Create Destination button and select the destination from the list.
    2. Select your account from the drop-down menu.
    3. Name your destination and click on Save to create your destination.
    Need to authorize another connection?

    Click on Add new Account in drop-down menu during authorizer selection and follow the on-screen prompts. You can also go to the Authorizers tab and click on Add New Service.

    Create a Flow to Azure Blob Storage

    1. On the Destinations page, click on the Create Destination button and select the destination from the list.
    2. Select your account from the drop-down menu.
    3. Name your destination and click on Save to create your destination.
    Need to authorize another connection?

    Click on Add new Account in drop-down menu during authorizer selection and follow the on-screen prompts. You can also go to the Authorizers tab and click on Add New Service.

    File Partitioning

    File partitioning refers to the practice of dividing large datasets into smaller, more manageable segments or partitions based on specific criteria, such as values in a particular column or range of values. Each partition contains a subset of the data that shares common attributes or characteristics. File partitioning is commonly used to improve data organization, query performance, and data management.

    Dataddo supports file partitioning during flow creation. If you use, for example,

    file_{{1d1|Ymd}}

    Dataddo will create a file every day, e.g. file_20xx0101, file_20xx0102 etc.

    Troubleshooting

    Server Failed to Authenticate the Request Error

    ERROR CODE

    stream transfer: write data from stream: write: uploading data to Azure Blob Storage: [0][AuthenticationFailed] Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature

    This error can be caused due to the following reasons:

    1. Invalid Shared Access Token
    2. Expired Shared Access Token
    3. Incorrect IP address restriction of Shared Access Token
    4. Invalid Container

    Invalid Shared Access Token Error

    1. Regenerate your Shared Access Token.
    2. Go to the Authorizers tab and click on your Azure Blob Storage authorizer.
    3. Insert the value of Blob SAS Token to the shared access token field and save the changes.

    Expired Shared Access Token Error

    1. Regenerate your Shared Access Token and make sure that the Expiry date of the token is long enough (we recommend at least 6 months).
    2. Go to the Authorizers tab and click on your Azure Blob Storage authorizer.
    3. Insert the value of Blob SAS Token to the shared access token field and save the changes.

    Incorrect IP Address Restriction of Shared Access Token Error

    1. Regenerate your Shared Access Token and make sure that Allowed IP addresses field is empty.
    2. Go to the Authorizers tab and click on your Azure Blob Storage authorizer.
    3. Insert the value of Blob SAS Token to the shared access token field and save the changes.

    Invalid Container Error

    1. Go to the Authorizers tab and click on your Azure Blob Storage authorizer.
    2. Make sure that the container name is correct.

    No Such Host Error

    ERROR CODE

    stream transfer: write data from stream: write: sending data: Put "https://address.com": dial tcp: lookup something.blob.core.windows.net on 0.0.0.0: no such host

    This error is caused by incorrect storage account.

    1. Go to the Authorizers tab and click on your Azure Blob Storage authorizer.
    2. Make sure that the storage account is correct.

    Was this article helpful?
    What's Next