In this article, we will discuss various date functions in PySpark

First, let's create a sample data frame:

from pyspark.sql import SparkSession
from pyspark.sql.functions import *

students = [{'id': '001', 'name': 'Achyut', 'dob': '1985-01-25'},
               {'id': '002', 'name': 'Chinmaya', 'dob': '1985-02-11'},
               {'id': '003', 'name': 'Ashmik', 'dob': '1985-02-02'},
               {'id': '004', 'name': 'Savar', 'dob': '2021-04-12'},
               {'id': '005', 'name': 'Trinabh', 'dob': '1985-01-25'}
               ]

students_df = spark.createDataFrame(students)
students_df.show()

Date Functions

All the pyspark functions are present in the package from pyspark.sql.functions

Reference : https://spark.apache.org/docs/2.4.2/api/python/pyspark.sql.html#module-pyspark.sql.functions

from pyspark.sql.functions import *

year()

This method is used to return the year from the given date.

students_df.select(
    col("dob"),
    year(col("dob")).alias("year")
).show()

month()

This method is used to return the month from the given date.

students_df.select(
    col("dob"),
    month(col("dob")).alias("month")
).show()

datediff()

This method is used to return the number of days between the current date and the given date.

students_df.select(
    current_date(), 
    col("dob"),                     
    datediff(current_date(), col("dob")).alias("days_difference")
).show()

months_between ()

This method is used to return the number of months between the current date and given date.

students_df.select(
    current_date(), 
    col("dob"), 
    months_between(current_date(), col("dob")).alias("months_difference")
).show()

add_months

This method is used to add months to the given date.

students_df.select(
    col("dob"),
    add_months(col("dob"),5).alias("after_5_months")
).show()

date_add

This method is used to add days to the given date.

students_df.select(
    col("dob"),
    date_add(col("dob"), 5).alias("after_5_days")
).show()

date_sub()

This method is used to subtract days to the given date.

students_df.select(
    col("dob"),
    date_sub(col("dob"), 5).alias("before_5_days")
).show()

---

If you like my work and want to support me…

1. I share tips, tricks and insights on #softwareengineering, #dataengineering #cloud #ml on LinkedIn.

2. Do you want to connect with me I have started mentoring others for career and interviews at 𝐭𝐨𝐩𝐦𝐚𝐭𝐞.𝐢𝐨/𝐧𝐚𝐯𝐞𝐞𝐧𝐩𝐧