Introduction In this tutorial, we want to count the number of values of a specific column. In order to do this, we use the COUNT function. We have already created the table student with the following data: Example 1 We want to count the total number of students. In order...
Introduction In this tutorial, we want to drop duplicates from a PySpark DataFrame. In order to do this, we use the the dropDuplicates() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can work with Pyspark, we need...
Introduction In this tutorial, we want to select data from a table that satisfy a certain condition. In order to do this, we use the SELECT Statement with the WHERE Clause. We have already created the table student with the following data: Example 1 We want to select all columns...
Introduction In this tutorial, we want to select data from a table and sort the result set. In order to do this, we use the SELECT Statement with the ORDER BY Clause. We have already created the table student with the following data: Example 1 - Sort rows in ascending...
Introduction In this tutorial, we want to select a specific number of rows from a table by using a SQL Statement. In order to do this, we use the SELECT Statement with the LIMIT Clause. We have already created the table student with the following data: SQL Statement We want...
Introduction In this tutorial, we want to filter specific rows from a PySpark DataFrame based on specific conditions. In order to do this, we use the the filter() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can...
Introduction In this tutorial, we want to sort a PySpark DataFrame by specific columns. In order to do this, we use the the orderBy() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can work with Pyspark, we...
Introduction In this tutorial, we want to drop columns from a PySpark DataFrame. In order to do this, we use the the drop() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can work with Pyspark, we need...
Introduction In this tutorial, we want to add columns to a PySpark DataFrame. In order to do this, we use the the withColumn() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession from pyspark.sql.functions import lit, col, sum, when from...
