Azure Synapse Analytics: Introduction to Metadata Activity [Introduction to Synapse - Ep. 11]

In this Pragmatic Works training video, Mitchell Pearson introduces the Get Metadata activity in Azure Synapse Analytics and explains why it is one of the most useful activities available when building pipelines. Mitchell presents this feature as a strong foundation for learning how to build more dynamic, automated, and data-driven ETL processes. Rather than doing a deep dive into every design pattern, he focuses on giving viewers a practical introduction to how the activity works, what it can return, and how its output can be used inside other pipeline activities.

Mitchell begins by explaining that as pipeline development becomes more advanced, teams often move beyond simple one-step processes and start building solutions that are driven by metadata, control tables, file detection, and iterative logic. In many real-world scenarios, organizations need pipelines that can determine what work needs to be done, identify whether files are available, and pass that information into downstream activities or child pipelines. The Get Metadata activity is especially helpful in these cases because it allows pipeline developers to inspect connected data sources and retrieve useful file or object properties at runtime.

Why the Get Metadata Activity Matters

According to Mitchell, the Get Metadata activity is valuable not only because of its functionality, but also because it teaches important concepts about how Azure Synapse pipelines work dynamically. It helps users understand how to retrieve outputs from activities, how to read JSON responses, and how to pass those values into future steps in a pipeline.

He explains that dynamic pipelines are often necessary for automation. For example, an organization may need a parent pipeline to identify work that needs to be done and then pass that work into child pipelines in a structured and repeatable way. In other cases, developers may want to use a control table, process a list of files, or only continue if a specific file exists. The Get Metadata activity supports these kinds of patterns by exposing useful properties from datasets and linked storage.

Setting Up the Activity

Mitchell demonstrates the feature inside the Synapse workspace that has been used throughout the video series. Instead of building a new dataset, he reuses an existing dataset from the previous lesson. That dataset connects to a linked service pointing to an Azure Data Lake and references a specific file within a folder structure.

To begin, he creates a new pipeline in the Integrate hub and names it after the activity. He then drags the Get Metadata activity onto the design canvas. Like other Synapse activities, the activity displays a required configuration warning until the necessary settings are completed.

The main steps Mitchell walks through are:

• Create a new pipeline in the Integrate hub.
• Add the Get Metadata activity from the General category.
• Open the activity settings.
• Select an existing dataset or create a new one if needed.
• Choose the fields to retrieve from the connected item.

Available Metadata Fields

One of the most important parts of the demonstration is the Field List. Mitchell describes this as the real power of the Get Metadata activity because it lets users choose which properties they want to inspect. He highlights several available fields and gives examples of how they might be used in real ETL processes.

In his example, he selects the following fields:

• Last Modified – useful for identifying whether a file is newer than the last processed version.
• Exists – useful for checking whether a file is available before processing begins.
• Item Name – useful when iterating through files dynamically.
• Size – useful for validation and verification purposes.

Mitchell also explains practical business scenarios for these values. A team might continuously check whether a file has arrived before starting a load. Another team might compare a file’s last modified date against a control table to determine whether the file should be processed again. These are common ETL use cases, and the Get Metadata activity helps support them in a flexible way.

Running the Pipeline and Reviewing Output

After configuring the activity, Mitchell runs the pipeline using Debug. He prefers Debug during development because it allows immediate testing without requiring the pipeline to be published first. This makes it easier to experiment, review outputs, and troubleshoot logic before committing changes.

Once the activity runs successfully, Azure Synapse provides both input and output details. Mitchell notes that the input section is simple in this case because the activity only includes the dataset and selected fields. The output section, however, is much more informative. It returns values in JSON format, including the metadata properties requested in the field list.

He explains that the output is made up of key-value pairs and can be referenced later in the pipeline. For example, developers can use expressions such as the activity name followed by .output and the property name to retrieve values like the last modified date or existence of the file.

Using Dynamic Content and Activity Outputs

A key learning point in the video is how to reference metadata output in another activity. To demonstrate this, Mitchell briefly adds an Until activity and connects it to the Get Metadata activity. This connection is important because it establishes the dependency and makes the previous activity’s output available in the expression builder.

He explains that once activities are connected, Synapse makes the output properties accessible through dynamic content. For example, the expression for checking whether a file exists can be inserted directly into another activity’s configuration. This is one of the most practical uses of metadata-driven development because it allows pipelines to make decisions based on runtime information.

Mitchell points out that this newer expression-building experience is more user-friendly than older methods, since Synapse now provides selectable activity output elements instead of requiring developers to manually construct every reference from memory.

Final Thoughts

Mitchell closes the lesson by emphasizing that the Get Metadata activity is one of the first activities developers should learn in Azure Synapse Analytics. It introduces important concepts such as dynamic content, activity output references, and automation-friendly pipeline design. While the video stays introductory, it gives viewers a clear understanding of how metadata can drive smarter ETL workflows. For users who want to build more flexible and scalable pipelines, this activity is an excellent place to start.

 Don't forget to check out the Pragmatic Works' on-demand learning platform for more insightful content and training sessions on Azure Synapse and other Microsoft applications. Be sure to subscribe to the Pragmatic Works YouTube channel to stay up-to-date on the latest tips and tricks.