What Are Data Flows?

In the context of Azure Synapse Analytics, data flows are used to extract, transform, and clean data through a graphical user interface. This concept is similar to other technologies like Power BI Data Flows and SSIS but with the benefit of Apache Spark behind the scenes, allowing massive parallel processing for big data problems. The key benefit is that all transformations in a data flow are converted to Scala and executed on an Apache Spark cluster without the need for manual coding.

How Data Flows Work in Synapse

When you create a data flow, you're working within the Develop Hub of Azure Synapse Analytics. To begin, you can create a source for your data, which typically involves selecting data stored in Azure, such as in a data lake or Azure SQL Database. In the video, Mitchell demonstrates using internet sales data as an example.

• Source: The first step is to define your data source. You can select a dataset from the available storage or connect to other data stores within Synapse, although data flows are somewhat limited in terms of the stores they can connect to.
• Debugging Data Flows: Once the source is set up, you can turn on data flow debug, which allows you to preview the data as it undergoes transformation. This feature is useful for checking how data looks after being transformed (e.g., null values replaced, columns split, etc.).

Data Transformation Options in Data Flows

Once you have your source data, you can begin transforming the data to suit your needs. This is where the power of data flows becomes evident, as it allows users to apply a range of transformations without writing code.

• Derived Columns: This is one of the most common transformations. It allows you to create new columns based on existing data. For instance, you could replace null values in a column with a default value (e.g., “Not Applicable”).
• Join Data: Data flows allow you to join multiple sources together based on specific conditions, much like traditional ETL tools.
• Aggregation and Pivoting: You can aggregate data (e.g., sum, count, average) and even pivot or unpivot data for better analysis and reporting.
• Expression Builder: The Expression Builder in data flows provides an intuitive way to apply complex functions to your data, such as coalesce to handle null values. This feature eliminates the need for coding while providing the flexibility of Apache Spark’s capabilities.

Working with Data Flows and Pipelines

In Synapse, data flows are often used in conjunction with pipelines. While data flows handle the transformation and cleaning of the data, pipelines are used to run those transformations as part of an automated workflow.

• Data Flow in Pipelines: To execute a data flow, you need to incorporate it into a pipeline. This allows you to reuse the data flow across multiple pipelines and automate its execution. You can also schedule when these pipelines run and manage how the data is moved and processed.

Loading Data

Once the data is cleaned and transformed, the next step is to load it into a sink. Synapse supports various sinks, including Azure Blob Storage, SQL databases, and Azure Data Lake. While some options may be grayed out depending on your setup, you can still perform data transformations and load them to these destinations with minimal configuration.

Common Design Patterns in Data Flows

Mitchell also discusses a common design pattern when working with data flows in Synapse: creating a pipeline that copies data into a data lake when it is not supported directly in the data flow. After copying the data to the lake, you can then transform and clean it before loading it into its final destination.

Conclusion: Why Use Data Flows?

Data flows in Azure Synapse Analytics are a powerful tool for cleaning and transforming data in a code-free environment. By using the built-in graphical user interface and Apache Spark, users can handle big data problems efficiently. Data flows are ideal when you need to perform complex data transformations that pipelines alone cannot accomplish.

• Use Data Flows: When you need to transform and clean data before loading it to a destination.
• Use Pipelines: To automate and run your data flows within a larger data processing workflow.