Mount Azure Blob Storage to Azure Databricks Cluster

Hey Folks,

This is kinda painful and the docs aren’t 100% easy to follow for this; so I’m going to write this out mostly for myself.

Step 1:  Create a Databricks Workspace & Cluster

Step 2: Create an Azure Key Vault

Step 3: Create a secret scope here: https://YOUR-REGION.azuredatabricks.net#secrets/createScope (after having signed into the workspace from the azure portal).

Step 4: Create a secret using the Azure Key Vault CLI: az keyvault secret set –vault-name “YOURVAULTNAME” –name “YOUR-SECRET-KEY” –value “YOUR KEY VALUE” –query “[?n]|[0]”

Step 5:  Mount the storage:

dbutils.fs.mount(
  source = "wasbs://MYCONTAINER@MYACCOUNT.blob.core.windows.net",
  mount_point = "/mnt/MOUNTDIR",
  extra_configs = {"fs.azure.account.key.MYACCOUNT.blob.core.windows.net":dbutils.secrets.get(scope = "dacrook-secrets", key = "mykey")})

Check to see that you can access it:

df = spark.read.text("/mnt/MYMOUNT/MYFILE.json")
print(df.head())

And thats it! Good luck!