Search for a command to run...
Articles tagged with #data-engineering
What is Hive Metastore Legacy metadata store for tables and schemas Linked to single Databricks workspace Stores based info : table names, locations, schema Limitation No centralized security across workspaces No column level access control H...
Let’s take a real-world ETL workflow in Databricks on AWS to see how the Control Plane and Data Plane work together. Scenario: Data Processing Pipeline You are a Data Engineer at an e-commerce company. Your task is to process customer orders from an ...
Let's consider it is a "20 Node Cluster" Each Node (30 Cores , 128GB RAM For good throughput let's assign 5 CORES per EXECUTOR --executor-cores = 5 Should leave 1 core for Background activity (Hadoop/Yarn daemons) Number of cores available = 30 -...
Time-based data processing is a critical aspect of data engineering, and PySpark provides a rich set of functions to handle date and time efficiently. 1. Extracting Year from a Date Column Problem: Extract the year from the column event_date. Solutio...
INSERT OVERWRITE replaces the existing data in the table but preserves historical versions of the data. Unlike CREATE OR REPLACE TABLE (CRAS), which modifies the schema, INSERT OVERWRITE generally only replaces the data. Dataset Example Let's assu...
Auto loader supports both directory listing and file notification but COPY INTO only supports directory listing. Auto loader file notification will automatically set up a notification service and queue service that subscribe to file events from the ...
