Delta Sharing - Databricks-to-Databricks Protocol

Understanding Databricks-to-Databricks Delta Sharing

In this Pragmatic Works training recap, Zane Goodman, Data Engineer Trainer, explains how Databricks-to-Databricks Delta Sharing works and why it is an important feature for secure, governed collaboration across environments. His walkthrough focuses on sharing data and AI assets between Unity Catalog-enabled Databricks workspaces, whether those workspaces exist in the same organization or across different organizations.

Zane Goodman begins by framing Delta Sharing as a flexible platform for securely sharing data and AI assets outside a single environment. While some Delta Sharing methods allow external consumers to access shared data without using Databricks, this video focuses specifically on the Databricks-to-Databricks protocol. This approach is designed for scenarios where both the provider and recipient are using Databricks workspaces with Unity Catalog enabled.

What Makes Databricks-to-Databricks Delta Sharing Different

Zane Goodman highlights that this protocol unlocks capabilities not typically available in broader sharing approaches. Because it uses the Delta Sharing server built directly into Databricks, organizations gain stronger governance, easier setup, and better performance. It also supports cross-cloud flexibility, meaning recipient workspaces can operate in AWS, Azure, or GCP.

He emphasizes several benefits of this model:

• Secure sharing between Unity Catalog-enabled Databricks workspaces
• Support for sharing tables, notebooks, volumes, models, and other assets
• Built-in auditing, governance, and usage tracking
• No need for the recipient to manage access tokens
• Simplified authentication and identity handling through the Databricks platform

According to Zane Goodman, one of the biggest advantages is that the provider does not have to manage token distribution manually. Authentication, verification, and auditing are handled by the platform, reducing overhead while maintaining security.

Why This Matters for Meta Store Isolation

A major use case in the video is sharing across different Unity Catalog metastores. Zane Goodman explains that metastores sit at the top of the Unity Catalog hierarchy and are isolated from one another by default. That means data in one metastore cannot simply be accessed from another without an intentional sharing mechanism.

This makes Databricks-to-Databricks Delta Sharing especially valuable for organizations that:

1. Operate multiple metastores across regions or business units
2. Need secure sharing across internal environments
3. Want to collaborate with external organizations using Databricks

He notes that although his demonstration shares data across metastores within the same tenant, the implementation process is nearly the same when sharing with another organization.

How Zane Goodman Demonstrates the Setup Process

In the walkthrough, Zane Goodman creates a share from a Databricks workspace he administers. He starts by selecting data from an existing catalog and schema, choosing the diamonds and taxi tables. He also includes table history, showing that the provider can control more than just base table access.

He then demonstrates another powerful feature of this protocol by adding a notebook to the share. This reinforces one of his key points: Databricks-to-Databricks sharing supports a broader range of assets than open sharing scenarios.

The core setup process follows these steps:

1. Create a new share in the Databricks workspace
2. Select the data assets to include
3. Optionally add notebooks or other supported assets
4. Create a recipient object
5. Enter the recipient sharing identifier
6. Grant the share to that recipient

Zane Goodman carefully explains that a recipient does not directly represent a single person. Instead, it points to the target metastore or receiving environment. To configure this, the provider needs the sharing identifier from the recipient side.

Recipient Access and Required Permissions

Once the share is created and assigned, the recipient side must mount that share to a catalog before users can work with it. Zane Goodman spends significant time clarifying the permission model because this is often where confusion happens.

He explains that a privileged user is required to create a catalog from the share. That privileged user could be:

• A metastore admin
• A user with both Create Catalog and Use Provider privileges
• A user with catalog creation rights and ownership of the provider object

Through his demo, Zane Goodman shows both the admin path and the more granular permission-based path. He also warns that granting catalog creation rights too broadly can introduce governance risks. For many organizations, the safer model is to have an admin mount the share first and then grant downstream access to the appropriate users.

What the Recipient Can Access

After mounting the share, the recipient can access shared tables and other assets through the new catalog. In Zane Goodman’s example, the shared notebook appears under Other Assets and opens in view-only mode. This is an important governance feature because recipients can consume the shared content without altering the source asset. If needed, they can clone the notebook for editing.

Final Takeaway

Zane Goodman closes by reinforcing that Databricks-to-Databricks Delta Sharing is a strong solution for secure, governed collaboration across isolated environments. His demonstration shows that the protocol is not only useful for cross-organization sharing, but also for internal scenarios where multiple metastores must exchange data safely.

For teams using Unity Catalog, this approach offers a practical balance of security, flexibility, governance, and usability. As Zane Goodman suggests, it is also just the beginning, with additional topics such as open sharing, partitioning, audit logs, and volume sharing offering even more depth for future exploration.

 Don't forget to check out the Pragmatic Works' on-demand learning platform for more insightful content and training sessions on Delta Sharing and other Microsoft applications. Be sure to subscribe to the Pragmatic Works YouTube channel to stay up-to-date on the latest tips and tricks.