Home / / Getting Started with Lakeflow Spark Declarative Pipelines (ETL Pipelines)
Getting Started with Lakeflow Spark Declarative Pipelines (ETL Pipelines)
Learn what they are, set up your environment, create ETL pipelines, transform data with views, schedule runs, and use row tracking and incrementalization for efficient processing.
In Getting Started with Lakeflow Spark Declarative Pipelines, Zane Goodman introduces a modern way to build ETL workflows in Databricks using a more declarative, structured approach. You’ll learn what Lakeflow Spark Declarative Pipelines are, why they matter, and how they help simplify data engineering by letting you define what the pipeline should produce while the platform handles more of the orchestration behind the scenes. This course is designed to help you understand the core concepts before jumping into hands-on pipeline creation.
You’ll set up the required environment, create an ETL pipeline, and work through common transformation patterns using views and structured data logic. Zane also covers scheduling so your pipelines can run automatically, along with row tracking and incrementalization to help process only what has changed instead of reprocessing everything from scratch. By the end, you’ll have a clear foundation for building Lakeflow Spark Declarative Pipelines that are cleaner, more maintainable, and better suited for scalable data workflows.
Course Outline ( Free Preview)
Module 00 - Introduction
Zane introduces the course and explains what you’ll learn about Lakeflow Spark Declarative Pipelines. You’ll get a high-level roadmap of the workflow, from understanding the concept to building, transforming, scheduling, and optimizing pipelines. This module sets the stage so you know how each step fits into a complete ETL process.
Module 01 - What are Lakeflow Spark Declarative Pipelines?
Learn what Lakeflow Spark Declarative Pipelines are and how they differ from more manual ETL approaches. Zane explains the value of defining pipeline logic declaratively, where the platform manages more of the execution and dependency flow. You’ll leave with a clear mental model for when this approach makes sense and why it can simplify data engineering work.
Module 02 - Setup11 min.
Before building a pipeline, you’ll get the necessary environment and resources in place. Zane walks through the setup requirements, including workspace readiness, access, and any configuration needed to follow along. This module helps prevent common blockers so you can focus on the pipeline build instead of troubleshooting setup issues.
Module 03 - Creating an ETL Pipeline14 min.
Start building your first Lakeflow Spark Declarative Pipeline. Zane demonstrates how to create the pipeline structure, define inputs and outputs, and understand the basic execution flow. You’ll learn how an ETL pipeline moves data from raw input toward a cleaner, more useful state for analytics and downstream processing.
Module 04 - Views and Transforming Data47 min.
Learn how views support transformation logic inside your pipeline. Zane shows how to shape, filter, and prepare data using views so each transformation step stays clear and reusable. This module focuses on building cleaner pipeline logic that is easier to understand, validate, and maintain over time.
Module 05 - Scheduling6 min.
Once your pipeline works, the next step is making it run automatically. Zane explains how scheduling fits into the pipeline lifecycle and how to think about timing, refresh cadence, and operational reliability. You’ll learn how to move from manual execution to a repeatable process that supports ongoing data workflows.
Module 06 - Row Tracking & Incrementalization22 min.
Finish by learning how row tracking and incrementalization make pipelines more efficient. Zane explains how incremental processing helps avoid unnecessary full reloads by focusing on new or changed data. You’ll see why this matters for performance, cost, and scalability as your data volumes and pipeline complexity grow.
Zane Goodman is a Trainer for Pragmatic Works specializing in the Power Platform. He's worked in skilled labor roles as well as the construction industry. Zane spent a lot of time in attics, on boom lifts, and in ditches making sure all the lights turned on properly. Now his primary focus is turning on the light for learning, helping our customers learn the ins and outs of Power Apps.
