Skip to main content

Command Palette

Search for a command to run...

Does Modifying a DataFrame Affect the View in PySpark?

Updated
โ€ข2 min read
Does Modifying a DataFrame Affect the View in PySpark?
N

I am a Tech Enthusiast having 13+ years of experience in ๐ˆ๐“ as a ๐‚๐จ๐ง๐ฌ๐ฎ๐ฅ๐ญ๐š๐ง๐ญ, ๐‚๐จ๐ซ๐ฉ๐จ๐ซ๐š๐ญ๐ž ๐“๐ซ๐š๐ข๐ง๐ž๐ซ, ๐Œ๐ž๐ง๐ญ๐จ๐ซ, with 12+ years in training and mentoring in ๐’๐จ๐Ÿ๐ญ๐ฐ๐š๐ซ๐ž ๐„๐ง๐ ๐ข๐ง๐ž๐ž๐ซ๐ข๐ง๐ , ๐ƒ๐š๐ญ๐š ๐„๐ง๐ ๐ข๐ง๐ž๐ž๐ซ๐ข๐ง๐ , ๐“๐ž๐ฌ๐ญ ๐€๐ฎ๐ญ๐จ๐ฆ๐š๐ญ๐ข๐จ๐ง ๐š๐ง๐ ๐ƒ๐š๐ญ๐š ๐’๐œ๐ข๐ž๐ง๐œ๐ž. I have ๐’•๐’“๐’‚๐’Š๐’๐’†๐’… ๐’Ž๐’๐’“๐’† ๐’•๐’‰๐’‚๐’ 10,000+ ๐‘ฐ๐‘ป ๐‘ท๐’“๐’๐’‡๐’†๐’”๐’”๐’Š๐’๐’๐’‚๐’๐’” and ๐’„๐’๐’๐’…๐’–๐’„๐’•๐’†๐’… ๐’Ž๐’๐’“๐’† ๐’•๐’‰๐’‚๐’ 500+ ๐’•๐’“๐’‚๐’Š๐’๐’Š๐’๐’ˆ ๐’”๐’†๐’”๐’”๐’Š๐’๐’๐’” in the areas of ๐’๐จ๐Ÿ๐ญ๐ฐ๐š๐ซ๐ž ๐ƒ๐ž๐ฏ๐ž๐ฅ๐จ๐ฉ๐ฆ๐ž๐ง๐ญ, ๐ƒ๐š๐ญ๐š ๐„๐ง๐ ๐ข๐ง๐ž๐ž๐ซ๐ข๐ง๐ , ๐‚๐ฅ๐จ๐ฎ๐, ๐ƒ๐š๐ญ๐š ๐€๐ง๐š๐ฅ๐ฒ๐ฌ๐ข๐ฌ, ๐ƒ๐š๐ญ๐š ๐•๐ข๐ฌ๐ฎ๐š๐ฅ๐ข๐ณ๐š๐ญ๐ข๐จ๐ง๐ฌ, ๐€๐ซ๐ญ๐ข๐Ÿ๐ข๐œ๐ข๐š๐ฅ ๐ˆ๐ง๐ญ๐ž๐ฅ๐ฅ๐ข๐ ๐ž๐ง๐œ๐ž ๐š๐ง๐ ๐Œ๐š๐œ๐ก๐ข๐ง๐ž ๐‹๐ž๐š๐ซ๐ง๐ข๐ง๐ . I am interested in ๐ฐ๐ซ๐ข๐ญ๐ข๐ง๐  ๐›๐ฅ๐จ๐ ๐ฌ, ๐ฌ๐ก๐š๐ซ๐ข๐ง๐  ๐ญ๐ž๐œ๐ก๐ง๐ข๐œ๐š๐ฅ ๐ค๐ง๐จ๐ฐ๐ฅ๐ž๐๐ ๐ž, ๐ฌ๐จ๐ฅ๐ฏ๐ข๐ง๐  ๐ญ๐ž๐œ๐ก๐ง๐ข๐œ๐š๐ฅ ๐ข๐ฌ๐ฌ๐ฎ๐ž๐ฌ, ๐ซ๐ž๐š๐๐ข๐ง๐  ๐š๐ง๐ ๐ฅ๐ž๐š๐ซ๐ง๐ข๐ง๐  new subjects.

No, modifying the original DataFrame after creating a view does not affect the view because views in PySpark are not directly linked to the DataFrame. Instead, the view stores the state of the DataFrame at the moment the view is created.

Create View and Modify DataFrame

  1. Creating a View and Modifying DataFrame
data = [("Ram", 30), ("Balaram", 25), ("Krishna", 35)]
columns = ["name", "age"]

df = spark.createDataFrame(data, columns)

# Create a temporary view
df.createOrReplaceTempView("people_view")

# Query the view
spark.sql("SELECT * FROM people_view").show()

Output

+-------+---+
|   name|age|
+-------+---+
|    Ram| 30|
|Balaram| 25|
|Krishna| 35|
+-------+---+
  1. Modify the Original DataFrame

     # Modify the original DataFrame (add a new column)
     df = df.withColumn("city", lit("New York"))
    
     # Show the modified DataFrame
     df.show()
    
     +-------+---+--------+
     |   name|age|    city|
     +-------+---+--------+
     |    Ram| 30|New York|
     |Balaram| 25|New York|
     |Krishna| 35|New York|
     +-------+---+--------+
    

Query the View Again

+-------+---+
|   name|age|
+-------+---+
|    Ram| 30|
|Balaram| 25|
|Krishna| 35|
+-------+---+

How to Update a View When DataFrame Changes?

If you want the view to reflect the updated DataFrame, you must recreate the view:

df.createOrReplaceTempView("people_view")  # Recreating the view

Now, if you query the view again:

spark.sql("SELECT * FROM people_view").show()

Output (Updated View after Recreation):

+-------+---+--------+
|   name|age|    city|
+-------+---+--------+
|    Ram| 30|New York|
|Balaram| 25|New York|
|Krishna| 35|New York|
+-------+---+--------+

Key Takeaways

More from this blog

Naveen P.N's Tech Blog

95 posts