In this video, Manuel Quintana discusses the recent updates and new features introduced in Fabric Dataflows Gen 2. With announcements made at a recent event, these updates aim to enhance the user experience and provide more robust capabilities for data engineering. These features include new sources, improved performance, cost reductions, and increased flexibility for ETL tasks.
Key Features of Dataflows Gen 2
- Improved Performance: Fabric Dataflows Gen 2 introduces a modern query evaluation engine that boosts performance, particularly with Azure Data Lake Gen 2, SharePoint files, and other sources.
- Preview Only Steps: This feature speeds up the design process by allowing users to evaluate transformations independently in the preview mode, ensuring faster development without affecting actual data loads.
- Partitioned Compute: By enabling parallel execution, this feature reduces evaluation time for large data sets, especially beneficial for users with partitioned files in Azure Data Lake Gen 2 and other sources.
- Cost Reduction: Dataflows Gen 2 now offers a more budget-friendly pricing model with reduced costs for long-running data flows and a flat rate for the first 10 minutes of query execution.
- Co-pilot Integration: Co-pilot, an AI tool integrated into Fabric, assists in creating calculated columns and explaining transformation steps, enhancing the user experience.
- Schema Support for Warehouses and Lakehouses: A new schema navigation feature improves the user experience when working with data from Azure SQL warehouses and Lakehouses, making it easier to explore hierarchical data.
- Incremental Refresh: The ability to set incremental refresh for large datasets improves efficiency by processing only the changes, significantly speeding up data refresh times.
Other Notable Updates
- New Destinations: Fabric Dataflows Gen 2 now supports additional destinations, including Snowflake, which will be available in preview shortly.
- Enhanced Integration with Power BI: For Power BI users, Dataflows Gen 2 offers an intuitive integration, allowing easy movement of data from various sources into Power BI's semantic model.
- Upcoming Events: The next major event for Fabric will be FabCon in Atlanta, March 2026, where additional updates and features will be announced.
Conclusion
Fabric Dataflows Gen 2 has received significant updates that improve performance, reduce costs, and enhance usability for ETL tasks. Whether you're new to dataflows or already using them, these changes provide valuable features for efficient and scalable data engineering. Keep an eye on future updates, as Fabric continues to evolve to meet the demands of enterprise-scale data workflows.
Don't forget to check out the Pragmatic Works' on-demand learning platform for more insightful content and training sessions on Fabric Dataflows and other Microsoft applications. Be sure to subscribe to the Pragmatic Works YouTube channel to stay up-to-date on the latest tips and tricks.