As a Data Engineer, I collect, extract and transform raw data in order to provide clean, reliable and usable data.
Introduction In order to work with data in Microsoft Fabric, you first need to get data into it. There are several ways to do this. One way is to manually upload files or folders from your local drive into a Fabric lakehouse. In this tutorial, we will explain step-by-step how...
Introduction Data transformation is a fundamental task in any data analysis or processing pipeline. In the realm of big data processing, Apache Spark has emerged as a powerful framework for handling large-scale data processing tasks efficiently. When it comes to transforming data within Spark, developers often have to choose between...
Introduction In the realm of big data processing, PySpark has emerged as a powerful tool for handling large-scale datasets. Its distributed computing framework allows for efficient processing of massive volumes of data. However, despite its capabilities, performing certain data transformations in PySpark can sometimes be cumbersome and complex. That'...
Introduction Data manipulation tasks often involve converting column data types to ensure consistency and accuracy in analysis. In this tutorial, we will show you how to change column types of a PySpark DataFrame. In order to do this, we will use the cast() function of PySpark. Import Libraries First, we...
Introduction Window functions in PySpark are a powerful feature for data manipulation and analysis. They allow you to perform complex calculations on subsets of data within a DataFrame, without the need for expensive joins or subqueries. In this tutorial, we will show you how to use window functions in PySpark....
Introduction FastAPI, a high-performance Python web framework, coupled with Docker, a powerful containerization tool, can significantly boost the efficiency of your development workflow. In this blog post, we'll walk you through the process of setting up a FastAPI project using a Dockerfile, providing a flexible and scalable solution...
Introduction One common task when working with large datasets is the need to generate unique identifiers for each record. In this tutorial, we will explore how to easily add an ID column to a PySpark DataFrame. In order to do this, we use the monotonically_increasing_id() function of PySpark....
Introduction One of the key tasks in data analysis is grouping data to gain insights and make informed decisions. In this tutorial, we will show you how to group the rows of a Pandas DataFrame and apply different aggregations on the grouped data. In order to do this, we will...
Introduction One of the key tasks in data analysis is grouping data to gain insights and make informed decisions. In this tutorial, we will show you how to group the rows of a PySpark DataFrame and apply different aggregations on the grouped data. In order to do this, we will...