
In this article, we will discuss various date functions in PySpark
First, let's create a sample data frame:
from pyspark.sql import SparkSession
from pyspark.sql.functions import *
students = [{'id': '001', 'name': 'Achyut', 'dob': '1985-01-25'},
{'id': '002', 'name': 'Chinmaya', 'dob': '1985-02-11'},
{'id': '003', 'name': 'Ashmik', 'dob': '1985-02-02'},
{'id': '004', 'name': 'Savar', 'dob': '2021-04-12'},
{'id': '005', 'name': 'Trinabh', 'dob': '1985-01-25'}
]
students_df = spark.createDataFrame(students)
students_df.show()

Date Functions
All the pyspark functions are present in the package from pyspark.sql.functions
Reference : https://spark.apache.org/docs/2.4.2/api/python/pyspark.sql.html#module-pyspark.sql.functions
from pyspark.sql.functions import *
year()
This method is used to return the year from the given date.
students_df.select(
col("dob"),
year(col("dob")).alias("year")
).show()

month()
This method is used to return the month from the given date.

students_df.select(
col("dob"),
month(col("dob")).alias("month")
).show()
datediff()
This method is used to return the number of days between the current date and the given date.
students_df.select(
current_date(),
col("dob"),
datediff(current_date(), col("dob")).alias("days_difference")
).show()

months_between ()
This method is used to return the number of months between the current date and given date.
students_df.select(
current_date(),
col("dob"),
months_between(current_date(), col("dob")).alias("months_difference")
).show()

add_months
This method is used to add months to the given date.
students_df.select(
col("dob"),
add_months(col("dob"),5).alias("after_5_months")
).show()

date_add
This method is used to add days to the given date.
students_df.select(
col("dob"),
date_add(col("dob"), 5).alias("after_5_days")
).show()

date_sub()
This method is used to subtract days to the given date.
students_df.select(
col("dob"),
date_sub(col("dob"), 5).alias("before_5_days")
).show()

If you like my work and want to support me…
I share tips, tricks and insights on #softwareengineering, #dataengineering #cloud #ml on LinkedIn.
Do you want to connect with me I have started mentoring others for career and interviews at 𝐭𝐨𝐩𝐦𝐚𝐭𝐞.𝐢𝐨/𝐧𝐚𝐯𝐞𝐞𝐧𝐩𝐧
About the author
I am a Tech Enthusiast having 13+ years of experience in 𝐈𝐓 as a 𝐂𝐨𝐧𝐬𝐮𝐥𝐭𝐚𝐧𝐭, 𝐂𝐨𝐫𝐩𝐨𝐫𝐚𝐭𝐞 𝐓𝐫𝐚𝐢𝐧𝐞𝐫, 𝐌𝐞𝐧𝐭𝐨𝐫, with 12+ years in training and mentoring in 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐓𝐞𝐬𝐭 𝐀𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐃𝐚𝐭𝐚 𝐒𝐜𝐢𝐞𝐧𝐜𝐞. I have 𝒕𝒓𝒂𝒊𝒏𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 10,000+ 𝑰𝑻 𝑷𝒓𝒐𝒇𝒆𝒔𝒔𝒊𝒐𝒏𝒂𝒍𝒔 and 𝒄𝒐𝒏𝒅𝒖𝒄𝒕𝒆𝒅 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 500+ 𝒕𝒓𝒂𝒊𝒏𝒊𝒏𝒈 𝒔𝒆𝒔𝒔𝒊𝒐𝒏𝒔 in the areas of 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭, 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠, 𝐂𝐥𝐨𝐮𝐝, 𝐃𝐚𝐭𝐚 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬, 𝐃𝐚𝐭𝐚 𝐕𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧𝐬, 𝐀𝐫𝐭𝐢𝐟𝐢𝐜𝐢𝐚𝐥 𝐈𝐧𝐭𝐞𝐥𝐥𝐢𝐠𝐞𝐧𝐜𝐞 𝐚𝐧𝐝 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠. I am interested in 𝐰𝐫𝐢𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐠𝐬, 𝐬𝐡𝐚𝐫𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐤𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞, 𝐬𝐨𝐥𝐯𝐢𝐧𝐠 𝐭𝐞𝐜𝐡𝐧𝐢𝐜𝐚𝐥 𝐢𝐬𝐬𝐮𝐞𝐬, 𝐫𝐞𝐚𝐝𝐢𝐧𝐠 𝐚𝐧𝐝 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 new subjects.
Comments
Join the discussionNo comments yet. Be the first to comment.


