When I’m working with Azure Data Factory, I often find some design tips that I like to share with the Azure community. If you’re working with Azure Data Factory, today I want to share a simple but important design feature when executing pipelines from within a pipeline.
In the screenshot below, you’ll see a pipeline that I created. This pipeline simply executes three other pipelines in sequence. I have created a task for a couple data warehouse loading tasks where I start with loading some lookups, followed by some dimension loads and finishing with fact tables. I want these 3 pipelines to run in sequence.
To set this up in Azure Data Factory, you just connect those pieces together but when I ran this each of those 3 pipelines ran at the same time instead of in sequence. So, here’s the design feature I want to point out. In the Execute Pipeline Activity is a setting called ‘Wait on Completion’. You can see where I set this option in the screenshot below.
It’s important to understand that this option is turned off by default. If you were trying to run pipeline activities in sequence, by default they will run asynchronously, meaning they’ll run without finishing before moving on.
This simple design feature of clicking that ‘Wait on Completion” box will enable one pipeline to execute only when one finishes.
So, I simple tip I thought was valuable to share. If you have any questions about Azure Data Factory, the Azure data platform or anything Azure related, you’re in the right place. Click the link below or contact us – we’re here to help no matter where you are on your Azure journey.