Databricks and Azure Key Vault

Jon Bloom Sep 11, 2020

In our ongoing Azure Databricks series within Azure Every Day, I’d like to discuss connecting Databricks to Azure Key Vault. If you’re unfamiliar, Azure Key Vault allows you to maintain and manage secrets, keys, and certificates, as well as sensitive information, which are stored within the Azure infrastructure.

In Azure Key Vault, we can maintain versioning over time and administer access to those keys within our organization. Databricks connect easily with Azure Key Vault, and I’ll walk you through it here. We will start with a scope and some secrets and then access them from Databricks.

I start with a Databricks stood up and our cluster is running.
In a recent post, I showed you how to connect to an Azure Storage Account from Databricks.
In my video demo, you’ll see the code that we use. The bad thing here is that we hard code many of these values and one of these is the key, which is sensitive as this is how we connect. We would not want to show this to most end users or developers as this would give them access into our storage account.
To get around this we can use the Key Vault. I created one in my demo and I have keys, secrets, and certificates in there.
When I go back to my notebook in Databricks I have the storage account name, the container, and the SAS token. In the Key Vault, I have the storage account container name, key value, and the account name.
In order to connect the Databricks to our Key Vault, you must create a scope and specify whether only the creator will have access or all users.

From this point, it is much clearer to show how to complete this connection and the code I used than to explain it, so please check out my video below where I’ll walk you through the process of connecting my Databricks to the Azure Key Vault.

There are many advantages for using Key Vault, not only with Databricks but a variety of other tools as well. If you’d like to learn more about how to leverage Azure Key Vault, Databricks or any Azure product or service, we’re here to help. Our team of Azure experts can help you in any stage of your Azure journey. Contact us or click below to learn more.