📘 Introduction Real-time data ingestion is a critical part of modern data architectures. Organizations need to process and store continuous streams of information for analytics, monitoring, and machine learning. Databricks, with the combined power of PySpark and Delta Lake, provides an efficient way to build end-to-end streaming pipelines that handle data...
📘 Introduction When working with large datasets in PySpark, joins can easily become performance bottlenecks. This happens because Spark needs to shuffle data across the cluster to match rows between DataFrames — a costly operation when both datasets are big. If one of your DataFrames is small, though, there’s a faster...
📘 Introduction In the age of data-driven decision making, a well-structured and scalable data architecture is essential. The Medallion Architecture is a proven framework that organizes data into multiple layers of refinement — ensuring clarity, governance, and trust as data flows from raw ingestion to business-ready insights. When combined with Databricks and...
Introduction In this tutorial, we will explain step-by-step how o read an Excel file into a PySpark DataFrame in Databricks. Configure Cluster First, install on a Databricks cluster the spark-excel library (also referred as com.crealytics.spark.excel). To do this, select your Databricks cluster in the "Compute"...
Introduction In this tutorial, we want to explain the characteristics of a Data Lakehouse. In order to do this, we will take a closer look at the key features of Data Lakes and Data Warehouses and how a Data Lakehouse combines the best of both worlds. Definition At its core,...