Why Use Azure Databricks?

There are many good reasons to use Azure Databricks. In this session of our mini-series on Azure Databricks, I’ll dig deeper into why you should use Databricks and the advantages that you’ll gain.

• With Databricks you’ll get the proprietary runtime improvement over Apache Spark. The originators created Spark, which started as Hadoop, and then the founders created the Databricks company. Then that progressed to Azure Databricks as a stand-alone component.
• You can process huge amounts of data with Databricks and since it is part of Azure, that data is cloud native. The data can be analyzed, processed, reported on, etc. all in the cloud. There are also many machine learning features to take advantage of.
• It keeps everything in memory along with better speed than other traditional methodologies.
• Clusters are easier to setup and configure.
• As it’s stored in the Azure cloud, it separates storage from compute. This saves you money as you are charged separately for compute vs storage, and the storage is fairly cheap. Also, when you shut down or delete a cluster, your data still lives in the cloud.

Benefits of Databricks:

• A key benefit is the tight integration into the Azure subscription.
  • You can integrate with the Azure Data Lake Store and Blob Storage to store, retrieve and update data.
  • Azure Data Factory can be used as part of your cloud-based extract, transform, load (ETL) process.
  • It has an Azure Synapse Analytics connector, as well as the ability to connect to Azure DB.
  • Integrates with your Active Directory.
  • Spin up a cluster using Azure DevOps and maintain your code in a source code repository. By using DevOps, you can save time by not being burdened by the administrative tasks that we’ve had in the past working with this type of data.
  • It has Ganglia to store your Metrics.
  • Databricks supports multiple languages. Scala is the main language, but it also works well with Python, SQL, and R.
  • You can have a collaborative notebook environment. Like Google Docs, people can comment in the margins of the notebook and those comments can be added in real time. There is also revision control to store revisions.
  • Databricks breaks down the silos between data engineers and data scientists, allowing each to be working on the same code at the same time throughout all the components of ELT, machine learning, etc. that you may integrate into your flow and process.

There are many ETL use cases in Azure Databricks such as genomics mapping, insurance, risk & regulation (fraud detection), IoT, and supply chain, among others.

Summary

Azure Databricks helps developers code quickly, in a scalable cluster, which is tightly integrated into Azure subscriptions. At the end of the day, you can extract, transform, and load your data within Databricks Delta for speed and efficiency. You can also ‘productionalize’ your Notebooks into your Azure data workflows.

If you would like to learn more about Azure Databricks and how to use it in your organization, our team of experts is here to help. Our goal is to show you how to use your data and Azure to grow your business. Contact us to learn more.