Choosing the Right Tools for ELT Workloads in the Cloud

Do you need help choosing the right tool for your cloud ELT workloads? This is a common question from clients who have existing ETL workloads on premise and are considering moving them into the cloud.

In most cases, clients have defined nightly batches that execute on Informatica or SSIS packages, etc. and they know how they get from point A to point B. What they want from me is to develop an architecture and tell them exactly what tools they would use when they migrate to the cloud.

Here’s my answer – it depends. It depends on the workload that you’re trying to do as we have a lot more tools in the cloud than we do on premise.

For all you golfers out there, let me equate this to playing a game of golf. In the modern ELT we have all sorts of tools; in golf, our tools are the 14 different clubs in our golf bag. Each club will hit the ball, but each is designed to do something different. Here’s my analogy:

• I start with a golf ball to move around the course, as well as a map of the course. The map tells me where I’m going to go, so I have a good predefined workload.
• My event is hitting the golf ball. Depending upon variables like where I am on the course, what I did prior and what I need to do next, I need to choose the right tool for the job.
• When I must hit the ball hard and far, I choose my driver. This could equate to an Azure Data Factory pipeline – a good, heavy hitting workload.
• Sometimes I don’t have to hit the ball quite as hard but require more precision. I equate that to maybe a set of stored procedures called by the Data Factory pipeline.
• If I’m very close to my destination, I have a single purpose club, my putter, to use when I need to have exact precision. I would equate my putter with the single purpose-built function of a Logic App.
• All my events get recorded on my scorecard which equates to an Azure Event Hub or maybe a logging table in your database.
• With my event hub or logging table (or scorecard in golf), I know that I’ve been able to complete my ELT workload (or game) using the tools that I need based upon the events throughout the whole course of play.

In conclusion, the tools that you’re going to use for your cloud ELT depends upon the event that you just did and what needs to happen next or in the game of golf, the tool used depends on the events that happened on the golf course.

I hope you found this helpful, whether you’re a golfer or not, and hopefully your ELT workloads in the cloud go as smooth as, or smoother in many cases, than your golf game. If you have more questions about ELT workloads in the cloud or about anything Azure related, we’d love to help. Click the link below or contact us – our experts are here to help no matter where you are on your cloud journey. Sorry, but we can’t be responsible to help with your golf game.