Leveraging PySpark with Delta and Magics in Microsoft Fabric
Getting Started with Real-Time Intelligence in Microsoft Fabric
Austin Libal introduces the concept of real-time intelligence within Microsoft Fabric, a comprehensive platform designed for managing data across various disciplines, including analytics, data engineering, and data science. In this session, Austin provides an overview of the real-time intelligence tools available in Microsoft Fabric and walks viewers through the process of working with real-time data streams.
What is Real-Time Intelligence?
Real-time intelligence allows organizations to monitor and respond to data as it is generated. Many industries, even those that don’t typically think of themselves as using real-time data, can benefit from real-time streams such as SQL databases that receive frequent updates. Real-time intelligence enables users to observe, analyze, and act on data in real-time, leading to faster insights and decision-making.
Microsoft Fabric Tools for Real-Time Intelligence
Microsoft Fabric provides several tools for managing real-time data. Some of these tools include:
- Event Streams
- KQL Databases (Kusto Query Language)
- Real-time dashboards
- Data Activator Reflex Items
Austin highlights how these tools enable users to work with data streams from SQL databases, Azure IoT devices, e-commerce sites, and more, using change data capture and other technologies.
Creating an Event Stream
Austin walks through the process of creating an event stream in Microsoft Fabric:
- Open the real-time intelligence persona and select the "Create Event Stream" option.
- Use a sample data set, such as bicycle data, to simulate real-time data streaming.
- Connect the data stream to a destination, such as a KQL database or a Lakehouse.
- Leverage enhanced capabilities (currently in preview) for a more user-friendly interface.
Using this setup, Austin demonstrates how data from sources like event hubs or IoT devices can be transformed and loaded into Microsoft Fabric for further analysis.
Working with KQL Databases
KQL (Kusto Query Language) databases are a powerful tool within Microsoft Fabric for analyzing massive amounts of data. Austin explains how KQL databases can handle petabytes of data efficiently, making them ideal for industries that generate large amounts of real-time data, such as e-commerce and IoT.
Loading Data to a KQL Database
Austin shows how to load real-time data into a KQL database:
- Create an event stream and set a destination, such as a KQL database.
- Create a new table in the KQL database to store the incoming data.
- Load data in JSON format into the table, which Austin demonstrates using a bicycle rental system example.
Querying Data in KQL
Once data is loaded into the KQL database, users can query it using the KQL language. Austin provides a few simple examples of querying the data:
- Selecting the first 100 records from the bicycle table
- Counting the total number of records in the table
- Filtering records by specific conditions, such as neighborhood
Additionally, Austin demonstrates how users who are more comfortable with SQL can write traditional SQL queries to retrieve and manipulate data in KQL databases.
Expanding Real-Time Intelligence
Looking ahead, Austin previews additional features of real-time intelligence, including creating KQL dashboards to monitor large datasets and integrating other data sources like Azure Event Hubs and IoT devices. Microsoft Fabric’s flexibility allows organizations to bring in data from multiple platforms, providing a holistic view of their operations in real-time.
Don't forget to check out the Pragmatic Works' on-demand learning platform for more insightful content and training sessions on Microsoft Fabric and other Microsoft applications. Be sure to subscribe to the Pragmatic Works YouTube channel to stay up-to-date on the latest tips and tricks.
Sign-up now and get instant access
ABOUT THE AUTHOR
Austin is a Jacksonville native who graduated from The Baptist College of Florida in 2012. He previously worked as a manager in the retail service industry. He enjoys spending time with his wife and two kids. His primary focus at Pragmatic Works is on Azure Synapse Analytics and teaching the best practices for data integration, enterprise data warehousing, and big data analytics. He also enjoys helping customers learn the ins and outs of Power BI and showing people new ways to grow their business with the Power Platform.
Free Trial
On-demand learning
Most Recent
private training
Leave a comment