Azure Data Factory Pipeline Templates

If you’re new to Azure Data Factory and unsure what you can do with it, I want to tell you about a new option within Data Factory called Pipeline Templates.

When you go to the Azure website, open the portal and go into the Data Factory Designer, there’s a new option on the ‘Let’s Get Started’ page for create a pipeline from a template. 

When you hit that button, it will take you to the template gallery, which has several tiles for different pipeline templates provided by Microsoft.

Again, if you’re new to Data Factory, this is a great way to get some ideas about what it can do and some of the services it can connect to. Also, if you’re using Data Factory for a certain technology, this is a way to learn about how to configure other technologies that you may not be using.

I think it’s always interesting to learn about the way other people are doing things. If you’ve been doing things a certain way, this may give you some other patterns or even technology to use in place of what you’re currently using. I also found some references to code, in particular some of the variable expressions and such that I found in the templates to use as a resource.

So, after I select one of the templates from the gallery, it will take me to configuration page. Look at my example below:

I've selected the copying data from Oracle to Azure SQL Data Warehouse. When I select this template, it gives me a preview of the activities that are included. In this case it’s a simple one where I just have a single activity in the template.

Then it’s asking me for a couple of inputs. Here it’s just looking at the two linked services that are required by the copy activity. I can select those from my drop down or I can create them like I would do with creating a link service outside of here.

So, I make a reference to my two connections, hit the button and it’s going to create a pipeline and it will be configured with those selections that I’ve made. It’s that simple.

Once you’ve created that pipeline from the template you could certainly modify the pipeline, but these templates are a good way to get started.

I also want to point out I can also get to the template gallery from the Designer. I have a new option under ‘Adding a Pipeline’ where I can select that I want to create a pipeline from a template. See below:

Another nice thing is we can create our own templates. So, if you have a pipeline and you’ve been copying it to maybe replicate the logic, etc., you can create your own template and make it available in the gallery. To do that you do need to be working within a Git repository, so you must have version control configured here.

If you’re working within your version control branch, you can create a pipeline and you’ll have an option (see screenshot below) that says Save as Template. This way once I get my pipeline working in the condition I want, I can save it and this template will now appear in the template gallery.

If I filter it down to show just my templates, you can see in my example here that I have one template created.

A nice feature here whether you’re new to Data Factory or some new options if you’re already a user. I hope this was helpful to you. If you have questions about Azure Data Factory, Data Warehousing in Azure or anything Azure related, our Azure experts can help, no matter where you are on your Azure journey. Click the link below or contact us.