As a Data Engineer, I collect, extract and transform raw data in order to provide clean, reliable and usable data.
Introduction In this tutorial, we will show you how to get a randomly sampled subset of a PySpark DataFrame. In order to do this, we will use the sample() function of PySpark. What is the sample() Function? The sample() function in PySpark is used to create a new DataFrame by...
Introduction In Microsoft Fabric, there are several activities that can be added to a data pipeline. One import activity is the Notebook activity. In this tutorial, we will explain step-by-step how to add a Notebook activity to a data pipeline using the Data Factory user interface. Goal A CSV File...
Introduction In Microsoft Fabric, there are several activities that can be added to a data pipeline. One import activity is the Copy activity. In this tutorial, we will explain step-by-step how to add a Copy activity to a data pipeline using the Data Factory user interface. Goal A CSV File...
Introduction In this tutorial, we will show you how to concatenate multiple string columns of a PySpark DataFrame into a single column. In order to do this, we will use the functions concat() and concat_ws() of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql...
Introduction In order to work with data in Microsoft Fabric, you first need to get data into it. There are several ways to do this. One way is to use a notebook. In this tutorial, we will explain step-by-step how to ingest data into a Fabric lakehouse using a notebook....
Introduction In this tutorial, we will show you how to remove the leading and trailing whitespaces from a string column of a PySpark DataFrame. In order to do this, we will use the functions trim(), ltrim() and rtrim() of PySpark. Import Libraries First, we import the following python modules: from...
Introduction In this tutorial, we will explain step-by-step how o read an Excel file into a PySpark DataFrame in Databricks. Configure Cluster First, install on a Databricks cluster the spark-excel library (also referred as com.crealytics.spark.excel). To do this, select your Databricks cluster in the "Compute"...
Introduction In order to work with data in Microsoft Fabric, you first need to get data into it. There are several ways to do this. One way is to use a dataflow. In this tutorial, we will explain step-by-step how to ingest data into a Fabric warehouse using a dataflow....
Introduction In order to work with data in Microsoft Fabric, you first need to get data into it. There are several ways to do this. One way is to use a data pipeline. In this tutorial, we will explain step-by-step how to ingest data into a Fabric warehouse using a...