In this second episode of the Azure Data Factory series, Austin Libal from Pragmatic Works delves into the concept of data transformations using the Select Transformation tool in Azure Data Factory. This transformation is part of the data flow process, allowing users to manipulate their data sets without the need to write complex code. Instead, a graphical user interface is used to define the changes. Below, we’ll walk through the key points from the video and highlight the steps involved in applying a select transformation to clean up data.
Key Steps in the Select Transformation Process
- Step 1: Setting up the Transformation - The process begins by opening Azure Data Factory and selecting the dataset you wish to transform. Austin demonstrates with a dataset of movie rankings from the website Letterboxd. If you haven't watched the previous video, it's recommended to review it first to understand how to bring in data and create datasets.
- Step 2: Adding a Select Transformation - After the data is brought in, Austin uses the Select schema modifier to remove unnecessary columns from the data set. This step helps to refine the dataset by focusing only on the relevant information.
- Step 3: Removing Unnecessary Columns - The Select Transformation is used to remove unwanted columns such as letterboxed URL and the date the movie was ranked. The goal is to keep only the essential columns like Movie Title, Year Released, and Rating.
- Step 4: Renaming Columns - Austin then renames columns for clarity. The Name column is renamed to Movie Title, and the Year column is renamed to Year Released for better readability.
- Step 5: Refreshing the Data Preview - After the transformations are applied, a quick data preview refresh is done to ensure that the changes have been successfully implemented. This allows the user to see a real-time result of the dataset after the transformation.
Benefits of Using Select Transformation in Azure Data Factory
- No Need for Code - The graphical user interface removes the need for complex SQL queries, enabling users to perform transformations easily without writing code. This is ideal for users who may not be familiar with SQL but still need to manipulate large datasets.
- Streamlined Data Cleanup - The Select Transformation allows for quick removal of unnecessary columns and reorganization of data, making the dataset cleaner and more focused.
- Enhanced Data Quality - By selecting only the necessary columns, the data becomes more manageable and suitable for further analysis or reporting.
Pragmatic Works Training Opportunities
If you're interested in learning more about Azure Data Factory or Data Flows, Pragmatic Works offers in-depth training opportunities through public boot camps and on-demand learning. Whether you're looking to dive deep into Azure Data Factory, or explore other Microsoft tools like Power BI or Power Automate, Pragmatic Works provides the resources you need to upskill and become more proficient with these technologies.
Conclusion
The Select Transformation in Azure Data Factory is an essential tool for cleaning up datasets and making them more manageable. By following the steps outlined in Austin’s tutorial, users can easily remove unwanted columns and rename data for clarity, ensuring the dataset is ready for further transformations or analysis. Stay tuned for more episodes in this series, as more advanced transformations will be explored in future videos.
Don't forget to check out the Pragmatic Works' on-demand learning platform for more insightful content and training sessions on Azure Data Factory and other Microsoft applications. Be sure to subscribe to the Pragmatic Works YouTube channel to stay up-to-date on the latest tips and tricks.