A list of premium posts for academy members containing hands-on tutorials, best practices, career advices and learning paths.
Introduction In this tutorial, we will explain step-by-step how o read an Excel file into a PySpark DataFrame in Databricks. Configure Cluster First, install on a Databricks cluster the spark-excel library (also referred as com.crealytics.spark.excel). To do this, select your Databricks cluster in the "Compute"...
Introduction In this tutorial, we will explain step-by-step how to query data from a Fabric warehouse using T-SQL. Goal Data should be queried from a table in the warehouse with T-SQL. Data We consider the table student that is stored in the warehouse "dlnerds_warehouse". Step 1: Open...
Introduction In order to work with data in Microsoft Fabric, you first need to get data into it. There are several ways to do this. One way is to use a dataflow. In this tutorial, we will explain step-by-step how to ingest data into a Fabric warehouse using a dataflow....
Introduction In order to work with data in Microsoft Fabric, you first need to get data into it. There are several ways to do this. One way is to use a data pipeline. In this tutorial, we will explain step-by-step how to ingest data into a Fabric warehouse using a...
Introduction One fundamental part of Microsoft Fabric is transforming data. Whether filtering, joining, merging or grouping data, there are several options available in Fabric to perform these operations. In this tutorial, we will explain step-by-step how to group data and apply an aggregation function using a dataflow. Goal A delta...
Introduction In this tutorial, we will show you how to group and concatenate strings in a PySpark DataFrame. In order to do this, we will use the groupBy() method in combination with the functions concat_ws(), collect_list() and array_distinct() of PySpark. Import Libraries First, we import the following...
Introduction In order to work with data in Microsoft Fabric, you first need to get data into it. There are several ways to do this. One way is to use a dataflow. In this tutorial, we will explain step-by-step how to ingest data into a Fabric lakehouse using a dataflow....
Introduction In order to work with data in Microsoft Fabric, you first need to get data into it. There are several ways to do this. One way is to use a data pipeline. In this tutorial, we will explain step-by-step how to ingest data into a Fabric lakehouse using a...
Introduction In order to work with data in Microsoft Fabric, you first need to get data into it. There are several ways to do this. One way is to manually upload files or folders from your local drive into a Fabric lakehouse. In this tutorial, we will explain step-by-step how...
Introduction Power BI makes it possible to bring numbers and information to life, tell stories and make data-driven decisions. This is the world of Power BI! Whether you are already familiar with data or just starting out, Power BI is a powerful tool that allows you to turn raw data...
Introduction Data transformation is a fundamental task in any data analysis or processing pipeline. In the realm of big data processing, Apache Spark has emerged as a powerful framework for handling large-scale data processing tasks efficiently. When it comes to transforming data within Spark, developers often have to choose between...
Introduction Microsoft Fabric is a powerful All-in-One Data Platform (SaaS) in the Azure Cloud that combines various Azure components to cover the fields of Data Integration, Data Engineering, Data Science and Business Intelligence. One key component of the Microsoft Fabric architecture is the Data Pipeline. In this tutorial, we will...
Introduction Microsoft Fabric is a powerful All-in-One Data Platform (SaaS) in the Azure Cloud that combines various Azure components to cover the fields of Data Integration, Data Engineering, Data Science and Business Intelligence. One key component of the Microsoft Fabric architecture is the Dataflow. In this tutorial, we will explain...
Introduction Microsoft Fabric is a powerful All-in-One Data Platform (SaaS) in the Azure Cloud that combines various Azure components to cover the fields of Data Integration, Data Engineering, Data Science and Business Intelligence. One key component of the Microsoft Fabric architecture is the Notebook. In this tutorial, we will explain...
Introduction Microsoft Fabric is a powerful All-in-One Data Platform (SaaS) in the Azure Cloud that combines various Azure components to cover the fields of Data Integration, Data Engineering, Data Science and Business Intelligence. One key component of the Microsoft Fabric architecture is the Warehouse. In this tutorial, we will explain...
Introduction Microsoft Fabric is a powerful All-in-One Data Platform (SaaS) in the Azure Cloud that combines various Azure components to cover the fields of Data Integration, Data Engineering, Data Science and Business Intelligence. One key component of the Microsoft Fabric architecture is the Lakehouse. In this tutorial, we will explain...
Introduction Microsoft Fabric is a powerful All-in-One Data Platform (SaaS) in the Azure Cloud that combines various Azure components to cover the fields of Data Integration, Data Engineering, Data Science and Business Intelligence. One key component of the Microsoft Fabric architecture is the Workspace. In this tutorial, we will explain...
Introduction Microsoft Fabric is a powerful All-in-One Data Platform (SaaS) in the Azure Cloud that combines various Azure components to cover the fields of Data Integration, Data Engineering, Data Science and Business Intelligence. In order to explore and get to know Fabric, Microsoft offers a free trial. In this tutorial,...
Introduction In the realm of big data processing, PySpark has emerged as a powerful tool for handling large-scale datasets. Its distributed computing framework allows for efficient processing of massive volumes of data. However, despite its capabilities, performing certain data transformations in PySpark can sometimes be cumbersome and complex. That'...
Introduction Power BI offers with the Power Query Editor a powerful tool for cleaning and transforming data. One important part of data preparation is adding an Index Column. For organizing and structuring your data it is crucial that every row is uniquely identified by an ID. An Index Column enables...
Introduction Power BI comes with the powerful formula language Data Analysis Expression (DAX) which allows the implementation of custom calculations. There are numerous operators and functions available in DAX. One essential type of DAX functions are aggregation functions. These functions are essential for data analysis, especially when it comes to...
Introduction As projects grow in size and complexity, it becomes increasingly important to ensure that code remains understandable and easy to work with. One powerful tool for achieving this is the use of type hints. In this tutorial, we will explain why and how to use type hints in Python....
Introduction Autoencoders, a cornerstone of unsupervised learning, have emerged as a powerful tool in the domain of artificial intelligence. By compressing and then reconstructing input data, these neural networks uncover meaningful representations, paving the way for applications ranging from image denoising to anomaly detection. In this tutorial we want to...
Introduction The first step when creating a Power BI report is to connect with data sources. Power BI can connect to a wide range of data sources. This capability allows users to access and analyze data from various sources within Power BI. One important format to often deal with is...
Introduction Traditional RNNs, limited by their simplistic structure, have problems retaining information over longer time periods, leading to the infamous vanishing gradient problem. Long Short-Term Memory (LSTM) Networks have the impressive ability to capture and preserve long-term dependencies in sequential data. But how is an LSTM able to do this?...
Introduction Environment variables are used for securely storing and accessing sensitive data, facilitating seamless configuration management across different environments. In this tutorial, we will explore how to work with environment variables in Python. In order to do this, we will use the Python libraries os and python-dotenv. What is an...
Introduction Power BI offers with the Power Query Editor a powerful tool for cleaning and transforming data. One important part of data preparation is filtering your data. Filtering enables you to sort out irrelevant data and to reduce the amount of data. One important type of filtering is custom filtering....
Introduction Window functions in PySpark are a powerful feature for data manipulation and analysis. They allow you to perform complex calculations on subsets of data within a DataFrame, without the need for expensive joins or subqueries. In this tutorial, we will show you how to use window functions in PySpark....
Introduction In Machine Learning, one essential step is evaluating the performance of a model. For classification models, the Confusion Matrix serves as a fundamental instrument for evaluating the performance. The Confusion Matrix provides a visualization of the results of a model. Based on the information from the Confusion Matrix, some...
Introduction FastAPI, a high-performance Python web framework, coupled with Docker, a powerful containerization tool, can significantly boost the efficiency of your development workflow. In this blog post, we'll walk you through the process of setting up a FastAPI project using a Dockerfile, providing a flexible and scalable solution...
Introduction In Machine Learning, one essential step is evaluating the performance of a model. For classification models, the Confusion Matrix serves as a fundamental instrument for evaluating the performance. It provides a clear and visual summary of the prediction accuracy of a model by illustrating the correspondence between the predicted...
Introduction One common task when working with large datasets is the need to generate unique identifiers for each record. In this tutorial, we will explore how to easily add an ID column to a PySpark DataFrame. In order to do this, we use the monotonically_increasing_id() function of PySpark....
Introduction In the fast-paced world of software development, efficiency and consistency are key. Docker, a powerful containerization platform, has revolutionized the way we build, ship, and run applications. In this tutorial, we show you how to get started with Docker. Step 1: Install Docker (if not already installed) First of...
Introduction PostgreSQL is one of the most widely used database management systems. One of the easiest ways to use PostgreSQL on macOS is the Postgres.app. Postgres.app provides a simple interface for setting up a server and a command-line interface (psql) for interacting with databases via the terminal. In...
Introduction In 1997, Sepp Hochreiter and Jürgen Schmidhuber introduced Long Short-Term Memory (LSTM), a special type of Recurrent Neural Network (RNN). The motivation was to overcome the problems of traditional RNNs when dealing with long sequences. This is exactly what LSTM Networks (LSTMs) are supposed to do. The special architecture...
Introduction In the landscape of software development, Docker has emerged as a game changer, offering a streamlined approach to building, shipping, and running applications. In this blog post, we explain the fundamental concept of Docker focusing on Dockerfiles, Docker images, and containers. What is Docker? Docker is an open-source platform...
Introduction One of the key tasks in data analysis is grouping data to gain insights and make informed decisions. In this tutorial, we will show you how to group the rows of a PySpark DataFrame and apply different aggregations on the grouped data. In order to do this, we will...
Introduction Data comes in different forms, each with its own characteristics and challenges. Basically, there are three main categories of data: Structured, Semi-structured and Unstructured Data. In this tutorial, we explore the characteristics and some examples for each kind of data. Structured Data First, let's have a look...
Introduction FastAPI has quickly gained popularity as a modern, fast and easy-to-use Python web framework for building RESTful APIs. In this tutorial, we show you step-by-step how to set up a FastAPI project. Prerequisites First of all, make sure you have Python installed on your system. Furthermore, it is recommended...
Introduction Power BI is one of the most widely used BI tools. But using Power BI on the Mac can be a challenge. This is because Microsoft does not offer a version of Power BI Desktop for the Mac. Nevertheless, there are workarounds for using Power BI on the Mac....
Introduction In this tutorial, we show you step-by-step how to set up a Python virtual environment on your Mac. Why Use a Virtual Environment? A virtual environment allows you to create isolated environments for different Python projects. This helps to avoid conflicts between project dependencies and ensures that each project...
Introduction In this tutorial, we want to explain the characteristics of a Data Lakehouse. In order to do this, we will take a closer look at the key features of Data Lakes and Data Warehouses and how a Data Lakehouse combines the best of both worlds. Definition At its core,...
Introduction In this tutorial we will cover how to represent boolean functions with the McCulloch-Pitts Neuron. We look at the boolean functions AND, OR and NOT. McCulloch-Pitts Neuron We have already explained how the McCulloch-Pitts Neuron works in another post. If you haven't seen it yet, check it...
Introduction In this tutorial, we want to explode arrays into rows of a PySpark DataFrame. In order to do this, we use the explode() function and the explode_outer() function of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession from pyspark.sql.functions...
Introduction In this tutorial, we want to explain why Generative AI (GenAI) is possible now. In order to do this, we describe the key factors that are responsible for the rise of Generative AI. Factors The key factors that enable Generative AI are availability of large datasets, computational power, and...
Introduction In this tutorial, we want to scale features of a Pandas DataFrame. In order to do this, we use the StandardScaler() class and the MinMaxScaler() class of Scikit-Learn. Import Libraries First, we import the following python modules: import pandas as pd from sklearn.datasets import fetch_california_housing from...
Introduction In this tutorial we will cover the very first and the simplest mathematical neuron model in history - the McCulloch-Pitts Neuron. We look at the architecture and functionality. History The McCulloch-Pitts-Neuron is the simplest form of a neuron model and was published in 1943 by Warren McCulloch and Walter...
Introduction In this tutorial, we want to one-hot encode a NumPy array that contains categorical values. In order to do this, we use the to_categorical() function of Keras. Import Libraries First, we import the following python modules: import numpy as np from keras.utils import to_categorical Define Data...
Introduction Machine Learning can be divided into two main types: Supervised Learning and Unsupervised Learning. In this tutorial, we want to take a closer look to these approaches and compare them to each other. Overview Both supervised learning and unsupervised learning have their own characteristics and are suitable for solving...
Introduction In this tutorial, we want to take a closer look to the most important Machine Learning algorithms and compare them to each other. Overview...
Introduction In contrast to a classic feedforward network, a Recurrent Neural Network (RNN) allows backward connections. An RNN is particularly suitable for processing time series data and is able to take into account the time dependency of data. This enables an RNN to have a kind of short-term memory. The...
Introduction In this tutorial, we want to import Stock Prices from Yahoo Finance into Python. In order to do this, we use ticker module of YFinance. YFinance The python library yfinance enables access to financial data from Yahoo Finance. Yahoo Finance provides various financial market data such as stock...
Introduction Data Scientists, Data Engineer and Data Analyst are some of the most familiar roles in the data world. In this tutorial, we want to take a closer look at each role . We look at their required skills and the technologies being used. Data Scientist...
Introduction In this tutorial, we want to explain how Generative Adversarial Neural Networks (GANs) work. In order to do this, we take a look at the functionality and the idea behind GANs. Principle Generative Adversarial Neural Networks (GANs) are a special type of Artificial Neural Networks. This kind of model...
Introduction Artificial Intelligence (AI) is developing at a rapid pace. Especially in the gaming industry, AI is now superior to humans. In this tutorial we will have a look at the games in which the AI is superior. Throughout history, there have been some groundbreaking achievements of AI. The successes...
Introduction In this tutorial, we want to explain Generative AI (GenAI). In order to do this, we describe both the terms Artificial Intelligence, Machine Learning, Deep Learning and Generative AI as well as the relationships between them. Overview First, we want to have a look to the relationships between Artificial...
Introduction In this tutorial, we want to create an Artificial Neural Network (ANN) that is able to recognize the species of penguins based on their body measurements. We will cover the entire process from Data Collection, Data Cleansing and Feature Engineering to Model Creation and Model Training. Use Case Based...
Introduction In this tutorial, we want to use regular expressions (regex) to filter, replace and extract strings of a PySpark DataFrame based on specific patterns. In order to do this, we use the rlike() method, the regexp_replace() function and the regexp_extract() function of PySpark. Import Libraries...
Introduction In this tutorial, we want to create a UDF and apply it to a PySpark DataFrame. In order to do this, we will show you two different ways: using the udf() function and using the @udf decorator. Import Libraries First, we import the following python modules: from pyspark.sql...
Introduction In this tutorial, we want to make aggregate operations on columns of a PySpark DataFrame. In order to do this, we use different aggregate functions of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession from pyspark.sql.functions import * Create SparkSession Before...
Introduction In this tutorial, we want to concatenate multiple PySpark DataFrames. In order to do this, we use the the union() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can work with Pyspark, we need to create...
Introduction In this tutorial, we want to join PySpark DataFrames. In order to do this, we use the the join() method of PySpark. Import Libraries First, we import the following python modules: from pyspark.sql import SparkSession Create SparkSession Before we can work with Pyspark, we need to create a...