Data Engineer

Data Engineer

As a Data Engineer, I collect, extract and transform raw data in order to provide clean, reliable and usable data.

59 posts
PySpark - Count Rows and Columns of a DataFrame
Academy Membership PySparkPython

PySpark - Count Rows and Columns of a DataFrame

Introduction In data processing and analysis with PySpark, it's often important to know the structure of your data, such as the number of rows and columns in a DataFrame. This is crucial for various operations, including data validation, transformations, and general exploration. In this tutorial, we'll...

PySpark - Count Distinct Values of a DataFrame Column
Academy Membership PySparkPython

PySpark - Count Distinct Values of a DataFrame Column

Introduction In this tutorial, we want to count the distinct values of a PySpark DataFrame column. In order to do this, we use the distinct().count() method and the  countDistinct() function of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession from pyspark.sql....

PySpark - How to create and use Broadcast Variables
Academy Membership PySparkPython

PySpark - How to create and use Broadcast Variables

Introduction In distributed computing environments like Apache Spark, efficient data handling is critical for performance. One useful feature for optimizing computations is broadcast variables. Broadcast variables allow you to share large read-only data across all nodes in a Spark cluster without duplicating the data for each task. In this tutorial,...

How to load a Sample Dataset for Real-Time Intelligence into an existing KQL Database in Microsoft Fabric

How to load a Sample Dataset for Real-Time Intelligence into an existing KQL Database in Microsoft Fabric

Introduction Microsoft Fabric offers a gallery of sample datasets that can be used to perform and practice Real-Time Analytics. These sample data sets offer a great opportunity to gain experience and familiarize yourself with the technologies and services in the Real-Time Intelligence Experience. In this tutorial we will show you...

You’ve successfully subscribed to Deep Learning Nerds | The ultimate Learning Platform for AI and Data Science
Welcome back! You’ve successfully signed in.
Great! You’ve successfully signed up.
Success! Your email is updated.
Your link has expired
Success! Check your email for magic link to sign-in.