“The auto storage account keys are invalid, please sync auto storage keys.” Have you seeing this error when trying to deploy application packages to Azure Batch? I did and here’s the good news: there’s a way to fix that.

The issue at hand

How can we apply the appropriate egress rules to the Azure Storage account firewall from an Azure Batch account? Note that this is for the Azure Batch front-end or user interface and not to the nodes themselves. We don’t want to allow access from ‘All networks’ and because we don’t know what IP address the Batch account will be using this is a little tricky and it required some investigation. I’ve bumped into this scenario with one of my partners recently so I’ve decided to document this experience in here.

In the portal, you’d see this error message:

storage error

Using azcli, we can actually a little more details:

Azure Error: AutoStorageKeysInvalid
Message: The auto storage account keys are invalid, please sync auto storage keys.
RequestId:f72bd21a-26db-4a14-bbab-f473830e896c
Time:2020-01-02T23:45:23.9454141Z
Target: BatchAccount

Ok. Let’s search for AutoStorageKeysInvalid

As I was searching for an answer to this, I’ve bumped into a ![post}(https://social.msdn.microsoft.com/Forums/en-US/50a4d99d-4c65-4091-8320-eb835ea300f2/package-deployment-failures?forum=azurebatch) on the Microsoft Developer Network that is very close to my issue. As mentioned in one of the answers from msrini:

One work around is to place the storage account in a different region(lets say Region A) other than the batch account(lets say Region B). In that case we can have the list of ips used by the batch service in that region B and add that to firewall in storage account. This will work for small application packages. But it will take more time for large packages.

And this is exactly what I will try. In a nutshell, for this to work you will have to have the Storage Account created on a different region than the service that is trying to access it. Because of that (being on different locations), we can use the information available on the configured public IP range on the location and use that to configure the firewall rules on the Storage Account.

storage-acls

Pre-reqs

Before you dive into the steps bellow, please take a note on the following assumptions:

  1. I’m running this on Ubuntu with bash.

  2. Download the file that contains all of the IP address ranges for Public Azure. This will have the information on what range belongs to what service (e.g.: Storage, SQL and Traffic Manager). This file is updated weekly and as of this writing it was last updated on 2019.12.16. You can get the file from here

azure pip

  1. Create a sample application package (just a .zip file so we can test).

    touch file.zip
    
  2. Set the environment variables that are specific to our my tests.

    export SUBSCRIPTION=XYZ
    export BATCH_RG=rg-batch-test
    export BATCH_NAME=batch-test
    export BATCH_APP_NAME=firewall-test
    export STORAGE_RG=firewall-test
    export ACCOUNT_NAME=fwtestacct
    

Small utilities to help the job

I’ve created a couple of small utils to help me with my tests. You can skip them if you want but they are here to facilitate your job in case you have to troubleshoot these rules and don’t want to retype these commands (they are huge) all of the time. Here they are:

  • create_batch will create an application package through an Azure Batch Account. It is important that when testing these settings, that you create not only the application itself (under Batch), but also that you upload an application package.

    create_batch()(az batch application package create --resource-group ${BATCH_RG} --name ${BATCH_NAME} --application-name ${BATCH_APP_NAME} --version-name 1 --subscription ${SUBSCRIPTION} --package-file file.zip)
    
  • delete_batch will remove the application file (.zip) as well as the application entry on Batch (which is actually a container on Blob Storage).

    delete_batch()(az batch application package delete --resource-group ${BATCH_RG} ${BATCH_NAME} --application-name ${BATCH_APP_NAME}
    --version-name 1 --subscription ${SUBSCRIPTION} --yes && az batch application delete --resource-group ${BATCH_RG} ${BATCH_NAME} --application-name ${BATCH_APP_NAME} --subscription ${SUBSCRIPTION}  --yes)
    
  • az-storage-add-network-rule will add an IP address to the firewall allowing traffic from it (in this example, from the Azure Batch front-end).

    az-storage-add-network-rule()(az storage account network-rule add --resource-group ${STORAGE_RG} --account-name ${STORAGE_ACCOUNT} --subscription ${SUBSCRIPTION} --action Allow  --ip-address ${1:-$(</dev/stdin)})
    
  • az-storage-remove-network-rule

    az-storage-remove-network-rule()(az storage account network-rule remove --resource-group ${STORAGE_RG} --subscription ${SUBSCRIPTION} --account-name ${STORAGE_ACCT} --ip-address ${1:-$(</dev/stdin)})
    

Export the functions so we can use it later on with xargs.

export -f az-storage-add-network-rule
export -f az-storage-remove-network-rule

Exporting the IP addresses

From here, you can select the IP addresses for a region and add them to the Azure Storage firewall. I’m going to be using jq to carve out the IP addresses for the Azure Batch service in the South Central US region. I will save its output into a file called acls.txt.

jq  '.values | .[] | select(.name == "BatchNodeManagement.SouthCentralUS") | .properties.addressPrefixes' ServiceTags_Public_20191216.json | awk -F '"' /,/'{print $2}' | sed 's/\/32//g' | tee acls.txt
13.65.192.161
13.65.208.36
20.45.122.224/27
23.101.176.33
23.102.178.148
23.102.185.64
70.37.49.163
104.214.19.192/27

Finally, enable the ACLs on all of the IP addresses.

cat acls.txt | xargs -I {} bash -c 'az-storage-add-network-rule "{}"'

adding rules en masse

At this point, you should be able to upload a file to your Batch account and that file should now find it’s way into the storage account. Let’s validate this with by uploading a zip file once again into the storage account.

$ create_batch
Uploading man.zip to storage blob https://fwtestacct.blob.core.windows.net/app-firewall-test...
{
"etag": "W/\"0x8D78FC19877A585\"",
"format": "zip",
"id": "/subscriptions/XYZ/resourceGroups/firewall-test/providers/Microsoft.Batch/batchAccounts/computernodes/applications/firewall-test/versions/1",
"lastActivationTime": "2020-01-02T20:23:14.511036+00:00",
"name": "1",
"resourceGroup": "firewall-test",
"state": "Active",
"storageUrl": "https://fwtestacct.blob.core.windows.net/app-firewall-test-...",
"storageUrlExpiry": "2020-01-03T00:23:14.838376+00:00",
"type": "Microsoft.Batch/batchAccounts/applications/versions"
}
command ran in 4.275 seconds.

Awesome! The file is there and the firewall is only allowing access from known IP addresses in Azure that are coming from Azure Batch in the specific location of my services.

Cleaning up

To remove a list of IP addresses in one shot you can do this:

cat acls.txt | xargs -I {} bash -c 'az-storage-remove-network-rule "{}"'

Hope this helps.