New Feature That Helps Manage Large Volumes of Data

Businesses today generate large volumes of data. How do we manage and govern all that data so we can accurately analyze it for critical business decisions?

Here I’ll show you how to do that in Azure, integrated with the Power BI service. This is a brand-new feature in Azure called Incremental Refresh Policies which is in Preview. Let’s take a look:

• First, I enable this preview feature in Power BI Desktop. This is designed to work in Power BI Premium capacity.
• In the Query Editor, I define two parameters called Range Start and Range End and I give them default values for development. (They must be Date/Time parameters.)
• I create a filter using ‘is after or equal to’ Range Start and ‘is before’ Range End.
• What that does is it translates to a native query using query folding.
• Since this is a SQL Server data source, we’re looking at the T-SQL generated here with a date range predicate that causes this query to only load a narrow band of data.
• It doesn’t matter if I perform steps that break query folding later because I’ve already partitioned my data.
• Next, I turn on an Incremental Refresh Policy on this table.
• I set it to say I’m going to load 5 years of historical data and one month of fresh data every time a data refresh kicks off.
• It’s not going to reload that 5 years of data, it’s only going to refresh the last month. Then at the end of year it will adjust the historical partition.
• But partitioning happens automatically, and it will load very large volumes of data in the Power BI Service, but I don’t have to upload and refresh all that data every time.

So, a great new feature for managing and governing all these huge volumes of data that businesses are taking in. You can focus on analyzing that data for your critical business decisions.

If you’d like to learn more about the Incremental Refresh Policies feature or anything Azure related, you’ve come to the right place. Click the link below or contact us – our Azure experts are here to help you take your business from good to great.