Skip to main content

Date Functions in PySpark

Updated 2 min readMarkdown
Date Functions in PySpark

In this article, we will discuss various date functions in PySpark

First, let's create a sample data frame:

from pyspark.sql import SparkSession
from pyspark.sql.functions import *

students = [{'id': '001', 'name': 'Achyut', 'dob': '1985-01-25'},
               {'id': '002', 'name': 'Chinmaya', 'dob': '1985-02-11'},
               {'id': '003', 'name': 'Ashmik', 'dob': '1985-02-02'},
               {'id': '004', 'name': 'Savar', 'dob': '2021-04-12'},
               {'id': '005', 'name': 'Trinabh', 'dob': '1985-01-25'}
               ]

students_df = spark.createDataFrame(students)
students_df.show()

Date Functions

All the pyspark functions are present in the package from pyspark.sql.functions

Reference : https://spark.apache.org/docs/2.4.2/api/python/pyspark.sql.html#module-pyspark.sql.functions

from pyspark.sql.functions import *

year()

This method is used to return the year from the given date.

students_df.select(
    col("dob"),
    year(col("dob")).alias("year")
).show()

month()

This method is used to return the month from the given date.

students_df.select(
    col("dob"),
    month(col("dob")).alias("month")
).show()

datediff()

This method is used to return the number of days between the current date and the given date.

students_df.select(
    current_date(), 
    col("dob"),                     
    datediff(current_date(), col("dob")).alias("days_difference")
).show()

months_between ()

This method is used to return the number of months between the current date and given date.

students_df.select(
    current_date(), 
    col("dob"), 
    months_between(current_date(), col("dob")).alias("months_difference")
).show()

add_months

This method is used to add months to the given date.

students_df.select(
    col("dob"),
    add_months(col("dob"),5).alias("after_5_months")
).show()

date_add

This method is used to add days to the given date.

students_df.select(
    col("dob"),
    date_add(col("dob"), 5).alias("after_5_days")
).show()

date_sub()

This method is used to subtract days to the given date.

students_df.select(
    col("dob"),
    date_sub(col("dob"), 5).alias("before_5_days")
).show()


If you like my work and want to support me…

  1. I share tips, tricks and insights on #softwareengineering, #dataengineering #cloud #ml on LinkedIn.

  2. Do you want to connect with me I have started mentoring others for career and interviews at 𝐭𝐨𝐩𝐦𝐚𝐭𝐞.𝐢𝐨/𝐧𝐚𝐯𝐞𝐞𝐧𝐩𝐧

About the author

  • I am a Tech Enthusiast having 13+ years of experience in 𝐈𝐓 as a 𝐂𝐨𝐧𝐬𝐮𝐥𝐭𝐚𝐧𝐭, 𝐂𝐨𝐫𝐩𝐨𝐫𝐚𝐭𝐞 𝐓𝐫𝐚𝐢𝐧𝐞𝐫, 𝐌𝐞𝐧𝐭𝐨𝐫, with 12+ years in training and mentoring in 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐓𝐞𝐬𝐭 𝐀𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞. I have 𝒕𝒓𝒂𝒊𝒏𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 10,000+ 𝑰𝑻 𝑷𝒓𝒐𝒇𝒆𝒔𝒔𝒊𝒐𝒏𝒂𝒍𝒔 and 𝒄𝒐𝒏𝒅𝒖𝒄𝒕𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 500+ 𝒕𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝒔𝒆𝒔𝒔𝒊𝒐𝒏𝒔 in the areas of 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐂𝐥𝐨𝐮𝐝, 𝐃𝐚𝐭𝐚 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬, 𝐃𝐚𝐭𝐚 𝐕𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧𝐬, 𝐀𝐫𝐭𝐢𝐟𝐢𝐜𝐢𝐚𝐥 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞 𝐚𝐧𝐝 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠. I am interested in 𝐰𝐫𝐢𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐠𝐬, 𝐬𝐡𝐚𝐫𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐤𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞, 𝐬𝐨𝐥𝐯𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐢𝐬𝐬𝐮𝐞𝐬, 𝐫𝐞𝐚𝐝𝐢𝐧𝐠 𝐚𝐧𝐝 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 new subjects.

No comments yet. Be the first to comment.

More from this blog