Getting Started with Real-Time Intelligence in Microsoft Fabric

Getting Started with Real-Time Intelligence in Microsoft Fabric

Austin Libal introduces the concept of real-time intelligence within Microsoft Fabric, a comprehensive platform designed for managing data across various disciplines, including analytics, data engineering, and data science. In this session, Austin provides an overview of the real-time intelligence tools available in Microsoft Fabric and walks viewers through the process of working with real-time data streams.

What is Real-Time Intelligence?

Real-time intelligence allows organizations to monitor and respond to data as it is generated. Many industries, even those that don’t typically think of themselves as using real-time data, can benefit from real-time streams such as SQL databases that receive frequent updates. Real-time intelligence enables users to observe, analyze, and act on data in real-time, leading to faster insights and decision-making.

Microsoft Fabric Tools for Real-Time Intelligence

Microsoft Fabric provides several tools for managing real-time data. Some of these tools include:

• Event Streams
• KQL Databases (Kusto Query Language)
• Real-time dashboards
• Data Activator Reflex Items

Austin highlights how these tools enable users to work with data streams from SQL databases, Azure IoT devices, e-commerce sites, and more, using change data capture and other technologies.

Creating an Event Stream

Austin walks through the process of creating an event stream in Microsoft Fabric:

1. Open the real-time intelligence persona and select the "Create Event Stream" option.
2. Use a sample data set, such as bicycle data, to simulate real-time data streaming.
3. Connect the data stream to a destination, such as a KQL database or a Lakehouse.
4. Leverage enhanced capabilities (currently in preview) for a more user-friendly interface.

Using this setup, Austin demonstrates how data from sources like event hubs or IoT devices can be transformed and loaded into Microsoft Fabric for further analysis.

Working with KQL Databases

KQL (Kusto Query Language) databases are a powerful tool within Microsoft Fabric for analyzing massive amounts of data. Austin explains how KQL databases can handle petabytes of data efficiently, making them ideal for industries that generate large amounts of real-time data, such as e-commerce and IoT.

Loading Data to a KQL Database

Austin shows how to load real-time data into a KQL database:

1. Create an event stream and set a destination, such as a KQL database.
2. Create a new table in the KQL database to store the incoming data.
3. Load data in JSON format into the table, which Austin demonstrates using a bicycle rental system example.

Querying Data in KQL

Once data is loaded into the KQL database, users can query it using the KQL language. Austin provides a few simple examples of querying the data:

• Selecting the first 100 records from the bicycle table
• Counting the total number of records in the table
• Filtering records by specific conditions, such as neighborhood

Additionally, Austin demonstrates how users who are more comfortable with SQL can write traditional SQL queries to retrieve and manipulate data in KQL databases.

Expanding Real-Time Intelligence

Looking ahead, Austin previews additional features of real-time intelligence, including creating KQL dashboards to monitor large datasets and integrating other data sources like Azure Event Hubs and IoT devices. Microsoft Fabric’s flexibility allows organizations to bring in data from multiple platforms, providing a holistic view of their operations in real-time.

Don't forget to check out the Pragmatic Works' on-demand learning platform for more insightful content and training sessions on Microsoft Fabric and other Microsoft applications. Be sure to subscribe to the Pragmatic Works YouTube channel to stay up-to-date on the latest tips and tricks.