Introduction In this tutorial, we want to concatenate multiple PySpark DataFrames. In order to do this, we use the the union() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can work with Pyspark, we need to create...
Introduction In this tutorial, we want to join PySpark DataFrames. In order to do this, we use the the join() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can work with Pyspark, we need to create a...
Introduction In this tutorial, we want to replace null values in a PySpark DataFrame. In order to do this, we use the the fillna() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession from pyspark.sql.functions import mean Create SparkSession Before...
Introduction In this tutorial, we want to drop rows with null values from a PySpark DataFrame. In order to do this, we use the the dropna() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can work with...
Introduction In this tutorial, we want to calculcate the sum of values of a specific column. In order to do this, we use the SUM function. We have already created the table student with the following data: Example 1 We want to calculcate the sum of credit points of all...
Introduction In this tutorial, we want to count the number of unique values of a specific column. In order to do this, we use the COUNT function with the DISTINCT keyword. We have already created the table student with the following data: Example 1 We want to count the number...
Introduction In this tutorial, we want to count the number of values of a specific column. In order to do this, we use the COUNT function. We have already created the table student with the following data: Example 1 We want to count the total number of students. In order...
Introduction In this tutorial, we want to drop duplicates from a PySpark DataFrame. In order to do this, we use the the dropDuplicates() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can work with Pyspark, we need...
Introduction In this tutorial, we want to select data from a table that satisfy a certain condition. In order to do this, we use the SELECT Statement with the WHERE Clause. We have already created the table student with the following data: Example 1 We want to select all columns...
