In this introductory episode of the Data Flows Series, Austin Libal from Pragmatic Works walks viewers through the fundamentals of using Azure Data Factory Data Flows for ETL processes. The video sets the stage for a practical, beginning-to-end use case that focuses on preparing raw data for reporting and business intelligence. Rather than diving immediately into advanced transformations, Austin introduces the core setup required before any data shaping can happen, making this a strong starting point for beginners and a helpful refresher for experienced data professionals.
Austin explains that the purpose of the series is to demonstrate how data can be extracted from a raw source, transformed into a more useful structure, and loaded into a final destination for analysis. This process is commonly known as ETL, which stands for Extract, Transform, and Load. In this case, Azure Data Factory Data Flows are used as the transformation layer.
He emphasizes that Data Flows are becoming increasingly popular because they offer a low-code to no-code environment. This makes them accessible to a wide range of users, whether someone has decades of experience in data movement or is just getting started. The graphical interface simplifies the transformation process and reduces the need for deep coding knowledge.
A key point Austin makes is that Data Flows are not the only way to transform data in Azure Data Factory, but they are a powerful and user-friendly option. Their visual interface allows users to create logic for data transformation without needing to write large amounts of SQL or other programming languages. This makes them especially valuable for teams that want to accelerate development and improve accessibility across technical skill levels.
In this episode, Austin focuses on building the foundation for future transformations by preparing a new Azure Data Factory environment and connecting it to source data.
Austin walks through the initial configuration required before transformation logic can be added. The process includes the following:
To begin, Austin demonstrates how to create a linked service from the Manage hub in Azure Data Factory. He describes a linked service as essentially a connection string to a data source. For this example, he connects to Azure Data Lake Storage Gen2, which contains the source data for the series.
After selecting the appropriate subscription and storage account, he tests the connection to confirm that it is valid. Once the linked service is successfully created, he moves to the Author hub to define a dataset. The dataset points to a CSV file containing movie ratings data. During configuration, Austin selects the first row as the header and imports the schema so that the data structure is immediately available inside Azure Data Factory.
This step is important because the dataset acts as the blueprint for the incoming data. It tells Data Factory what file is being used, where it is located, and how its contents should be interpreted.
Once the linked service and dataset are created, Austin publishes the changes to save the new artifacts. He then creates a new data flow and gives it a meaningful name tied to the movie ratings example. This data flow will become the workspace where future transformation logic is built throughout the series.
Before exploring the data, he turns on Data Flow Debug. Austin explains that this feature spins up the compute resources needed to process and preview transformations. Specifically, it uses an Apache Spark cluster, which powers the transformation engine behind Data Flows.
One of the most useful features Austin highlights is the ability to inspect the data directly inside the data flow. After adding the movie ratings source, he reviews the projection, which displays the available columns and their data types. He also uses Data Preview to load a live sample of the dataset.
This preview allows users to see how their data looks before and after transformations are applied. Austin notes that whether the next step involves a select, filter, or aggregate transformation, users can validate their work visually in near real time. This helps improve accuracy, speeds up development, and makes troubleshooting much easier.
This first episode is designed as an introduction, not a deep transformation lesson. Austin closes by previewing the next topic in the series, which will focus on a select transformation to remove unnecessary columns from the dataset. That progression makes this episode especially valuable because it builds the technical foundation viewers need before moving into more advanced transformation logic.
Through this walkthrough, Austin Libal gives learners a clear and approachable introduction to Azure Data Factory Data Flows. For professionals interested in building reporting-ready datasets through visual ETL tools, this episode provides the right starting point. Pragmatic Works also encourages viewers to continue learning through its Azure Data Factory bootcamp, Azure Synapse Analytics training, and on-demand learning platform.
Don't forget to check out the Pragmatic Works' on-demand learning platform for more insightful content and training sessions on Azure Data Factory and other Microsoft applications. Be sure to subscribe to the Pragmatic Works YouTube channel to stay up-to-date on the latest tips and tricks.