Skip to main content

Apache Spark VS Databricks

Updated 3 min readMarkdown
Part of seriesData Engineering
Apache Spark VS Databricks

Apache Spark is one of the main data processing engines in data lake house architecture. Apache Spark provides speed, ease of use with wide range of use cases:

  • Data integration and ETL

  • Interactive Analytics

  • Realtime Streaming

  • Graph Parallel Computation

  • Machine learning and advanced analytics

But Spark lacks many essential features that needed real-time.

  1. ACID Transaction capabilities

  2. Metadata Catalog

  3. Cluster Management

  4. Automation APIs and Tools

  5. Data Storage Infrastructure

Databricks builds on top of Spark and created an eco-system that helps end to end solution architecture. Databricks is founded by the authors of Apache Spark. It’s a commercial product, but it has a free community edition with many features. Below are key features that Databricks brings to the table:

ACID Transactions via Delta Lake Integration

ACID transactions guarantee that each read, write, or modification of a table has the following properties:

Atomicity: Either the entire statement is executed, or none of it is executed.

Consistency: Consistency ensures that corruption or errors in your data do not create unintended consequences.

Isolation: When multiple users are reading and writing from the same table all at once, isolation of their transactions ensures that the concurrent transactions don't interfere with or affect one another.

Durability: Ensures that changes to your data made by successfully executed transactions will be saved, even in the event of system failure.

Unity Catalog for Metadata Management

  1. Unity Catalog offers a unified governance layer for data.

  2. With Unity Catalog, organizations can seamlessly govern their structured and unstructured data, machine learning models, notebooks, dashboards and files on any cloud or platform.

  3. Access management with a unified interface to define access policies on data.

Cluster Management

Databricks provides cluster management options including displaying, editing, starting, terminating, deleting, controlling access, and monitoring performance and logs. We can also use the Clusters API to manage compute programmatically.

Secure Cloud Storage Integration

Databricks uses cloud object storage to store data files and tables. During workspace deployment, Databricks configures a cloud object storage location known as the DBFS root. Databricks supports configuring connections to other cloud object storage locations.

  • Use Unity Catalog to connect and manage other cloud storage locations (Recommended way)

  • Mount and use other cloud storage locations.

Notebooks and Workspace

Notebooks are the primary tool for creating data science and machine learning workflows and collaborating with colleagues. Databricks notebooks provide real-time coauthoring in multiple languages, automatic versioning, and built-in data visualizations.

Photon Query Engine

Photon is a vectorized query engine written in C++ that leverages data and instruction-level parallelism available in CPUs.

It’s 100% compatible with Apache Spark APIs which means you don’t have to rewrite your existing code ( SQL, Python, R, Scala) to benefit from its advantages.

Photon is an ANSI compliant Engine, it was primarily focused on SQL but now the scope is much larger, with more ingestion sources, formats, APIs and methods since the launch.

Automation Tools

Databricks Workflows supports scheduling jobs, triggering them or having them run continuously when building pipelines for real-time streaming data. Databricks Workflows also provides advanced monitoring capabilities and efficient resource allocation for automated jobs.

About the author

  • I am a Tech Enthusiast having 13+ years of experience in 𝐈𝐓 as a 𝐂𝐨𝐧𝐬𝐮𝐥𝐭𝐚𝐧𝐭, 𝐂𝐨𝐫𝐩𝐨𝐫𝐚𝐭𝐞 𝐓𝐫𝐚𝐢𝐧𝐞𝐫, 𝐌𝐞𝐧𝐭𝐨𝐫, with 12+ years in training and mentoring in 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐓𝐞𝐬𝐭 𝐀𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞. I have 𝒕𝒓𝒂𝒊𝒏𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 10,000+ 𝑰𝑻 𝑷𝒓𝒐𝒇𝒆𝒔𝒔𝒊𝒐𝒏𝒂𝒍𝒔 and 𝒄𝒐𝒏𝒅𝒖𝒄𝒕𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 500+ 𝒕𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝒔𝒆𝒔𝒔𝒊𝒐𝒏𝒔 in the areas of 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐂𝐥𝐨𝐮𝐝, 𝐃𝐚𝐭𝐚 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬, 𝐃𝐚𝐭𝐚 𝐕𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧𝐬, 𝐀𝐫𝐭𝐢𝐟𝐢𝐜𝐢𝐚𝐥 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞 𝐚𝐧𝐝 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠. I am interested in 𝐰𝐫𝐢𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐠𝐬, 𝐬𝐡𝐚𝐫𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐤𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞, 𝐬𝐨𝐥𝐯𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐢𝐬𝐬𝐮𝐞𝐬, 𝐫𝐞𝐚𝐝𝐢𝐧𝐠 𝐚𝐧𝐝 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 new subjects.

More from this blog