As a Data Engineer, I collect, extract and transform raw data in order to provide clean, reliable and usable data.
Introduction One common task when working with large datasets is the need to generate unique identifiers for each record. In this tutorial, we will explore how to easily add an ID column to a PySpark DataFrame. In order to do this, we use the monotonically_increasing_id() function of PySpark....
Introduction One of the key tasks in data analysis is grouping data to gain insights and make informed decisions. In this tutorial, we will show you how to group the rows of a Pandas DataFrame and apply different aggregations on the grouped data. In order to do this, we will...
Introduction One of the key tasks in data analysis is grouping data to gain insights and make informed decisions. In this tutorial, we will show you how to group the rows of a PySpark DataFrame and apply different aggregations on the grouped data. In order to do this, we will...
Introduction FastAPI has quickly gained popularity as a modern, fast and easy-to-use Python web framework for building RESTful APIs. In this tutorial, we show you step-by-step how to set up a FastAPI project. Prerequisites First of all, make sure you have Python installed on your system. Furthermore, it is recommended...
Introduction In this tutorial, we want to write a PySpark DataFrame to a CSV file. In order to do this, we use the csv() method and the format("csv").save() method of PySpark DataFrameWriter. Besides, we use DataFrame.write for creating a DataFrameWriter instance. Import Libraries First, we...
Introduction In this tutorial, we want to explain the characteristics of a Data Lakehouse. In order to do this, we will take a closer look at the key features of Data Lakes and Data Warehouses and how a Data Lakehouse combines the best of both worlds. Definition At its core,...
Introduction In this tutorial, we want to read a CSV file into a PySpark DataFrame. In order to do this, we use the csv() method and the format("csv").load() method of PySpark DataFrameReader. Besides, we use spark.read for creating a DataFrameReader instance. Import Libraries First, we...
Introduction In this tutorial, we want to explode arrays into rows of a PySpark DataFrame. In order to do this, we use the explode() function and the explode_outer() function of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession from pyspark.sql.functions...
Introduction In this tutorial, we want to add the current date and the current timestamp to a PySpark DataFrame. In order to do this, we use the current_date() function and the current_timestamp() function of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import...