Skip to main content

Streaming Deduplication and Quality Enforcement

Updated 3 min readMarkdown
Streaming Deduplication and Quality Enforcement

This program demonstrates a real-time data pipeline using Spark Structured Streaming to handle deduplication and data quality enforcement on streaming data from CSV files.

Objective

The program achieves the following:

  1. Ingest data from a directory containing CSV files.

  2. Deduplicate records based on unique identifiers (event_id) and keep the latest record within a specified time window.

  3. Enforce data quality rules by filtering and handling invalid or missing data.

  4. Output processed data to the console for debugging and verification.

Code Walkthrough

Reading Streams

Spark Session Initialization

The program starts by initializing a SparkSession, which is the entry point for Spark Structured Streaming applications.

spark = SparkSession.builder \
    .appName("Streaming Deduplication and Quality Enforcement") \
    .master("local") \
    .getOrCreate()

Schema Definition

A custom schema is defined to specify the structure of the incoming CSV files:

CUSTOM_SCHEMA = StructType([
    StructField("event_id", StringType(), True),    # Unique event identifier
    StructField("name", StringType(), True),       # Name of the entity
    StructField("timestamp", TimestampType(), True),  # Event timestamp
    StructField("value", IntegerType(), True)      # Numeric value for the event
])

Reading Streaming Data

The program reads streaming data from a folder (../resources/dataset/data_quality/input) using the CSV format:

streaming_df = spark.readStream \
    .format("csv") \
    .schema(CUSTOM_SCHEMA) \
    .load("../resources/dataset/data_quality/input")

Deduplication Using Time-Based Windowing

Deduplication is achieved by grouping records using a time-based window and event_id. Records with the latest timestamp are kept.

Add Watermark and Grouping

streaming_df.withWatermark("timestamp", "10 minutes").groupBy(
    window(col("timestamp"), "10 minutes"),  # Group by 10-minute time windows
    col("event_id")                         # Group by event_id
)

Aggregate Functions

.agg(
    max("timestamp").alias("latest_timestamp"),  # Latest timestamp per group
    max("value").alias("value"),                # Latest value per group
    first("name").alias("name")                 # First non-null name
)

Data Quality Enforcement

Data quality is enforced by applying filters and handling null values:

quality_enforced_df = deduplicated_df \
    .filter((col("value").isNotNull()) & (col("value") > 0)) \
    .fillna({"name": "Unknown"})  # Replace null names

Writing to the Console

The processed data is written to the console using the writeStream method:

quality_enforced_df.writeStream \
    .format("console") \
    .outputMode("complete") \
    .start() \
    .awaitTermination()

Input Data

Input

event_id,name,timestamp,value
1,John,2024-12-10 10:00:00,50
1,John,2024-12-10 10:05:00,60
2,Alice,2024-12-10 10:03:00,30
3,,2024-12-10 10:10:00,-10

Output

+--------+-------------------+-----+-------+
|event_id|timestamp          |value|name   |
+--------+-------------------+-----+-------+
|1       |2024-12-10 10:05:00|60   |John   |
|2       |2024-12-10 10:03:00|30   |Alice  |
+--------+-------------------+-----+-------+

Complete Code

from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql.types import *

if __name__ == '__main__':
    # Create Spark session
    spark = SparkSession.builder \
        .appName("Streaming Deduplication and Quality Enforcement") \
        .master("local") \
        .getOrCreate()

    # Define custom schema
    CUSTOM_SCHEMA = StructType([
        StructField("event_id", StringType(), True),    # Unique event identifier
        StructField("name", StringType(), True),       # Name of the entity
        StructField("timestamp", TimestampType(), True),  # Event timestamp
        StructField("value", IntegerType(), True)      # Numeric value for the event
    ])

    # Source: Read streaming data from CSV files
    streaming_df = spark.readStream \
        .format("csv") \
        .schema(CUSTOM_SCHEMA) \
        .load("../resources/dataset/data_quality/input")

    # Deduplicate using time-based windowing
    deduplicated_df = streaming_df.withWatermark("timestamp", "10 minutes") \
        .groupBy(
            window(col("timestamp"), "10 minutes"),  # Group by time window
            col("event_id")                         # Group by event_id
        ) \
        .agg(
            max("timestamp").alias("latest_timestamp"),  # Keep the latest timestamp
            max("value").alias("value"),                # Keep the latest value
            first("name").alias("name")                 # Keep the first non-null name
        ) \
        .select(
            col("event_id"),
            col("latest_timestamp").alias("timestamp"),
            col("value"),
            col("name")
        )

    # Enforce data quality rules
    quality_enforced_df = deduplicated_df \
        .filter((col("value").isNotNull()) & (col("value") > 0)) \
        .fillna({"name": "Unknown"})  # Replace null names

    # Sink: Write the results to the console
    quality_enforced_df.writeStream.start(format="console",
                                          outputMode="complete").awaitTermination()

About the author

  • I am a Tech Enthusiast having 13+ years of experience in 𝐈𝐓 as a 𝐂𝐨𝐧𝐬𝐮𝐥𝐭𝐚𝐧𝐭, 𝐂𝐨𝐫𝐩𝐨𝐫𝐚𝐭𝐞 𝐓𝐫𝐚𝐢𝐧𝐞𝐫, 𝐌𝐞𝐧𝐭𝐨𝐫, with 12+ years in training and mentoring in 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐓𝐞𝐬𝐭 𝐀𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞. I have 𝒕𝒓𝒂𝒊𝒏𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 10,000+ 𝑰𝑻 𝑷𝒓𝒐𝒇𝒆𝒔𝒔𝒊𝒐𝒏𝒂𝒍𝒔 and 𝒄𝒐𝒏𝒅𝒖𝒄𝒕𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 500+ 𝒕𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝒔𝒆𝒔𝒔𝒊𝒐𝒏𝒔 in the areas of 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐂𝐥𝐨𝐮𝐝, 𝐃𝐚𝐭𝐚 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬, 𝐃𝐚𝐭𝐚 𝐕𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧𝐬, 𝐀𝐫𝐭𝐢𝐟𝐢𝐜𝐢𝐚𝐥 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞 𝐚𝐧𝐝 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠. I am interested in 𝐰𝐫𝐢𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐠𝐬, 𝐬𝐡𝐚𝐫𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐤𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞, 𝐬𝐨𝐥𝐯𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐢𝐬𝐬𝐮𝐞𝐬, 𝐫𝐞𝐚𝐝𝐢𝐧𝐠 𝐚𝐧𝐝 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 new subjects.

No comments yet. Be the first to comment.

More from this blog