Skip to main content

Without Unity Catalog in Databricks

Updated 4 min readMarkdown
Part of seriesData Engineering
Without Unity Catalog in Databricks

Before the introduction of Unity Catalog in Databricks, managing data and controlling access within the platform required different approaches. These methods, though effective in certain scenarios, lacked the unified governance, simplified access management, and centralized metadata handling that Unity Catalog now provides. Below is a summary of how things were done before Unity Catalog in the context of data governance, access control, and data management:

1. Data Governance Without Unity Catalog

Prior to Unity Catalog, data governance in Databricks involved managing governance policies manually across different data systems. Key governance tasks such as data lineage, metadata management, and auditing had to be handled through custom implementations and separate tools.

Manual Lineage Tracking:

  • Lineage refers to the tracking of the data’s origin and how it is transformed over time.

  • Before Unity Catalog, data lineage was manually tracked using external tools like Apache Atlas or custom scripts. These tools were not fully integrated with Databricks and required extensive configuration to map out how data flows across different jobs, pipelines, and transformations.

Decentralized Metadata Storage:

  • Metadata (e.g., table structures, schema definitions, and data types) had to be managed independently for each workspace. There wasn’t a centralized location to store and access metadata across Databricks environments.

  • Databricks workspaces would store metadata locally, often requiring users to maintain and update their own metadata catalogs or rely on external metadata repositories that were not always synchronized with the platform.

Compliance and Auditing:

  • Auditing and ensuring compliance with data policies and standards were done manually using logs and external tools.

  • There was no unified mechanism for tracking who accessed or modified data. Organizations had to configure their own logging mechanisms (often through Apache Spark or external systems) to keep track of user activities and changes to the data.


2. Access Control Before Unity Catalog

Without Unity Catalog, managing access to data in Databricks was done using a combination of workspace-level permissions, cluster configurations, and external tools.

Workspace-Level Permissions:

  • Users and groups were assigned permissions at the Databricks workspace level rather than the data level.

  • Permissions controlled access to the overall workspace but were not fine-grained enough for individual data objects, such as tables, schemas, or views. This made it challenging to manage who could access specific data assets within the workspace.

Cluster-Level Permissions:

  • Users who had cluster access could potentially access all data within the workspace, unless additional restrictions were applied.

  • This broad level of access made it harder to implement least-privilege access, where users are only given the minimum level of access necessary for their work.

External Identity and Permissions Management:

  • For managing user identities and access, Databricks integrated with external systems like Active Directory or OAuth. However, this integration was typically applied to workspace-level access rather than data-level access.

  • Managing fine-grained permissions on individual tables or views often required custom configurations and complex workflows.

3. Data Management Without Unity Catalog

Data management before Unity Catalog was often fragmented and required manual setup and integration.

Manual Organization of Data:

  • Data Organization in Databricks was based on workspaces and the DBFS (Databricks File System), where users would create directories and store data in files. Managing large volumes of data in this decentralized structure required additional tools and scripting.

  • There was no standard way to manage data objects like tables, views, and schemas across workspaces.

Limited Integration with External Data Sources:

  • Although Databricks supported integration with various data sources (e.g., Amazon S3, Azure Blob Storage, Delta Lake), the management and organization of this external data were not fully centralized. Each workspace would independently manage connections to external sources, making it harder to apply consistent governance or permissions across different sources.

Metadata Management:

  • Managing metadata was a separate task that required integration with external tools or custom scripts.

  • Without Unity Catalog’s centralized Metastore, metadata for tables, schemas, and other objects was often scattered across different workspaces or metadata repositories, requiring synchronization between various systems.

Challenges Before Unity Catalog:

  • Fragmented Governance: There was no single, unified framework to ensure consistent data governance across the platform. Different tools were needed for lineage tracking, auditing, and managing metadata, often requiring significant overhead to integrate and maintain.

  • Complex Access Management: Managing fine-grained data access at the level of individual tables, schemas, or data assets was difficult without Unity Catalog’s centralized access controls. Permissions had to be manually configured for each workspace or cluster, which increased the potential for errors and security risks.

  • Lack of Metadata Centralization: Storing and managing metadata across multiple workspaces was cumbersome. Users had to rely on external systems to manage data definitions, lineage, and schema information, leading to inconsistencies and challenges with synchronization.

Summary of Key Differences:

About the author

  • I am a Tech Enthusiast having 13+ years of experience in 𝐈𝐓 as a 𝐂𝐨𝐧𝐬𝐮𝐥𝐭𝐚𝐧𝐭, 𝐂𝐨𝐫𝐩𝐨𝐫𝐚𝐭𝐞 𝐓𝐫𝐚𝐢𝐧𝐞𝐫, 𝐌𝐞𝐧𝐭𝐨𝐫, with 12+ years in training and mentoring in 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐓𝐞𝐬𝐭 𝐀𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞. I have 𝒕𝒓𝒂𝒊𝒏𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 10,000+ 𝑰𝑻 𝑷𝒓𝒐𝒇𝒆𝒔𝒔𝒊𝒐𝒏𝒂𝒍𝒔 and 𝒄𝒐𝒏𝒅𝒖𝒄𝒕𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 500+ 𝒕𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝒔𝒆𝒔𝒔𝒊𝒐𝒏𝒔 in the areas of 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐂𝐥𝐨𝐮𝐝, 𝐃𝐚𝐭𝐚 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬, 𝐃𝐚𝐭𝐚 𝐕𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧𝐬, 𝐀𝐫𝐭𝐢𝐟𝐢𝐜𝐢𝐚𝐥 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞 𝐚𝐧𝐝 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠. I am interested in 𝐰𝐫𝐢𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐠𝐬, 𝐬𝐡𝐚𝐫𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐤𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞, 𝐬𝐨𝐥𝐯𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐢𝐬𝐬𝐮𝐞𝐬, 𝐫𝐞𝐚𝐝𝐢𝐧𝐠 𝐚𝐧𝐝 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 new subjects.

No comments yet. Be the first to comment.

More from this blog