As a Data Engineer, I collect, extract and transform raw data in order to provide clean, reliable and usable data.
Introduction Keeping the data fresh in your data warehouse is crucial. In dbt (data build tool), source freshness checks help ensure that the data coming from your sources is up-to-date and trustworthy, enabling teams to confidently build reliable models downstream. This is where defining source freshness in the sources.yml...
Introduction Once you've defined sources in dbt, you can reference them using the source() function. This is essential for dbt to identify dependencies between sources and models. In this hands-on tutorial, we’ll show you how to use the source() function with a sample project. 📌 This is a...
Introduction In dbt (data build tool), sources represent raw data tables that already exist in your data warehouse - often provided by upstream systems like operational databases, APIs or other applications. Defining these sources in sources.yml is a crucial step in your data project. This tutorial will guide you...
Introduction FastAPI is a modern, fast (high-performance) web framework for building APIs with Python. When building real-world applications, you’ll often need to manage sensitive information like API keys, database URLs, or feature toggles that should vary across environments (development, testing, production). Using environment variables and Pydantic’s BaseSettings is...
Introduction Seeds in dbt are a simple and powerful way to load static data - such as reference tables, lookup data, or mapping tables - directly into your data warehouse. In this tutorial, you'll learn what seeds are and how to use them effectively. Since this topic is...
Introduction Want to start with dbt core but don’t know where to begin? Don’t worry! In this tutorial, we'll walk through setting up a new dbt project from scratch - we cover the entire process from creating a virtual environment to initializing your project and verifying...
Introduction Modern data work doesn’t always require complex infrastructure. If you're looking for a fast, local-first analytics stack, dbt + DuckDB is a powerful combination. In this tutorial, we'll walk through connecting dbt to DuckDB — perfect for analysts, developers, and data enthusiasts who want simplicity without...
Introduction In dbt, the dbt_project.yml file is the backbone of your project configuration. It controls how dbt behaves, where it looks for different resource types, and how your models are materialized and organized. In this tutorial, we'll walk through a sample dbt_project.yml file and...
Introduction When working with PySpark DataFrames, understanding the statistical properties of your data is crucial for data exploration and preprocessing. PySpark provides the describe() and summary() functions to generate useful summary statistics. In this tutorial, we’ll explore how to use both functions to get insights into our dataset. 📥 Import...