In this video, Austin Libal, an expert from Pragmatic Works, demonstrates how to use the Filter Transformation inside Azure Data Factory (ADF) to filter data effectively in a no-code environment. Azure Data Factory is a powerful tool for managing and transforming data in the cloud without requiring extensive coding knowledge. With its graphical interface, users can manipulate data and create custom transformations for reporting purposes with ease.
Azure Data Factory’s data flows are designed to help users transform data without needing to write SQL queries or complex scripts. The Filter Transformation is particularly useful for filtering records based on certain criteria, making it ideal for use cases where users want to manipulate data based on specific conditions. In this video, Austin focuses on using the Filter Transformation to remove unwanted data—specifically, movies that have a rating lower than four stars in a movie ranking dataset.
To get started, Austin explains how to set up the data flow in Azure Data Factory. The first steps include importing the source dataset and removing unnecessary columns using the Select Transformation. If you're unfamiliar with how to bring in data or use the Select Transformation, Austin encourages viewers to check out previous videos in the series.
After setting up the source data and removing unwanted columns, the next step is to apply the Filter Transformation.
To apply the filter, Austin adds the Filter Transformation using the “plus” icon in the data flow interface. He names the transformation "Remove Movies Below Four Stars" to clearly describe the action being performed. This makes it easier to track what each transformation does, especially in more complex data flows.
Austin explains that the filter will be applied to the rating column of the movie dataset. The goal is to exclude movies with ratings lower than four stars, only keeping movies rated four or five stars.
The next step is using the Expression Builder, a key feature in Azure Data Factory. This tool allows users to build expressions without needing to know SQL. Austin demonstrates using the greater or equal function in the Expression Builder to filter the data.
The expression used is:
greaterOrEquals(rating, 4)
This ensures that only records with ratings of four or more are returned. Austin also highlights the use of IntelliSense, a helpful feature in the Expression Builder that provides code suggestions and helps users build expressions more efficiently.
Once the filter is applied, Austin encounters an issue where the rating column is in string format, but the comparison function requires an integer. This results in an error, and Austin demonstrates how to resolve this by adding a Cast Transformation.
The Cast Transformation is used to change the data type of the rating column from string to integer, making it compatible with the comparison operation. Austin explains how to select the column and specify the new data type, ensuring that the transformation works correctly.
After applying the Cast Transformation, Austin revisits the Filter Transformation and runs a data preview to verify the results. He checks the data to ensure that no movies with a rating lower than four stars are included.
Austin’s data preview successfully returns only movies with ratings of four or five stars, confirming that the filter has worked as intended. He concludes the demonstration by noting that the Filter and Cast Transformations can be combined to clean and manipulate data effectively in Azure Data Factory, even without extensive coding knowledge.
Austin emphasizes the power of low-code/no-code transformations in Azure Data Factory. By using a graphical interface, users can perform complex data manipulations without writing a single line of code. This approach is ideal for business users and analysts who may not have extensive programming experience but still need to transform data for reporting purposes.
In this video, Austin Libal walks through the process of using the Filter Transformation in Azure Data Factory to manipulate data without writing code. He shows how to filter data based on specific conditions and resolve common issues, such as data type mismatches, by using the Cast Transformation. This process is essential for users looking to refine their data for reporting, making Azure Data Factory an invaluable tool in the modern data workflow.
For those interested in learning more about Azure Data Factory and data flows, Austin encourages viewers to explore the on-demand learning platform offered by Pragmatic Works, where users can find more in-depth tutorials and training on these topics.
Don't forget to check out the Pragmatic Works' on-demand learning platform for more insightful content and training sessions on Azure Data Factory and other Microsoft applications. Be sure to subscribe to the Pragmatic Works YouTube channel to stay up-to-date on the latest tips and tricks.