Introduction

In this tutorial, we want to add the current date and the current timestamp to a PySpark DataFrame. In order to do this, we use the current_date() function and the current_timestamp() function of PySpark.

Import Libraries

First, we import the following python modules:

from pyspark.sql import SparkSession
from pyspark.sql.functions import current_date, current_timestamp

Create SparkSession

Before we can work with Pyspark, we need to create a SparkSession. A SparkSession is the entry point into all functionalities of Spark.

In order to create a basic SparkSession programmatically, we use the following command:

spark = SparkSession \
    .builder \
    .appName("Python PySpark Example") \
    .getOrCreate()

Create PySpark DataFrame

Next, we create the PySpark DataFrame with some example data from a list. To do this, we use the method createDataFrame() and pass the data and the column names as arguments.

column_names = ["language", "framework", "users"]
data = [
    ("Python", "Django", 20000), 
    ("Python", "FastAPI", 9000), 
    ("Java", "Spring", 7000), 
    ("JavaScript", "ReactJS", 5000)
]
df = spark.createDataFrame(data, column_names)
df.show()

Add Current Date

We would like to add the current date to the PySpark DataFrame.

To do this, we use the current_date() function of PySpark. The date will be in the format "yyyy-MM-dd":

new_df = df.withColumn("curr_date", current_date())
new_df.show()

Add Current Timestamp

We would like to add the current timestamp to the PySpark DataFrame.

To do this, we use the current_timestamp() function of PySpark. The timestamp will be in the format "yyyy-MM-dd HH:mm:ss.SSS":

new_df = new_df.withColumn("curr_timestamp", current_timestamp())
new_df.show()

Conclusion

Congratulations! Now you are one step closer to become an AI Expert. You have seen that it is very easy to add the current date and the current timestamp to a PySpark DataFrame. We can simply use the current_date() function and the current_timestamp() function of PySpark. Try it yourself!

Instagram

Also check out our Instagram page. We appreciate your like or comment. Feel free to share this post with your friends.