How to Use the Power Apps AI Builder Object Detector

AI Object Detection is a cool new feature in the Power Platform which can be accomplished using AI Builder. AI Builder allows you to add artificial intelligence to your apps even if you have no coding or data science skills.

In this post, I’ll show you how to use the AI Builder for object detection in Power Apps. In this app, I’ll use AI to automatically detect what’s in a picture and to ultimately do an inventory system.

For this app, I will use pictures of different soda cans in various places (a cooler or my desk, for instance). My goal is to build an app where an inventory person can take a picture of the soda and it would automatically add these items to inventory. Let’s get started!

• Prior to building my app, I had to create a Common Data Services (CDS) entity (which is required for AI Builder) that has a list of all the product SKUs I have (Coke, Diet Coke, Mountain Dew, etc.) and the inventory of those SKUs.
• Next, I go to make.powerapps.com, then click on AI Builder on the left side menu and then Build. The screen will show choices for Make Your Own AI Model (some are in preview).
• For this example, I click on Object Detection and give this AI model a name (Soda Inventory), then hit Create.
• It will ask what is the CDS entity I want to use. I click on object name and search for the table I previously created (with the list of all my inventory) which you can load from a CSV or Excel file.
• When I select Fields, I’ll get a list of the fields or objects I may want to do inventories on. In my case, I select the last four which are my sodas, then click Next.
• The next step is to upload images that have those objects pictured. Note, you’ll need to add 15 images at the bare minimum per object. It won’t even let you go to market unless you have at least 15 images; you really need about 40-50 to get a nice, crisp model.
• You also want to ensure that your can of Coke, for instance, is shown in various natural/real world settings for the model to perform well. For an inventory it may be on a shelf, in a box or even a trash can or desk, again in places where you’d naturally see it, not all light up with stage lighting, etc.
• Once those images are uploaded, I need to go through and tag each image. When you hover over the picture, if you wait for the ‘hand’ icon, object detection will give you a list of choices of what the image is showing (various sodas) and you can click on the one that applies to create the tag. A bit of a slow, painful process.
• In the case of Canada Dry Ginger Ale and Mountain Dew, which have similar green cans, I want to take many of those cans pictured side by side so the model can figure out that these are two different types of cans.
• When done, I click on Done Tagging and go to the next step which is to train the model.
• In my demo, I’ll go back to a model I’ve already created to see how it works. When I open my Soda Inventory Detection model, I’ll see a performance rating of 87. In my case, I can improve this model by adding more data. I only uploaded about 30 images, with 15-20 of each object.
• I click on quick test to see if this is going to work. I can click on any image and it detects and labels each soda in the picture which a percentage of how sure it is that it’s correct.
• Next, I’ll use this inside of my Power App to build an inventory system.
• I create a new Power App and click on Insert and AI Builder and choose Object Detection from that drop down.
• From here, I choose my model for my demo which is my Soda Inventory Detection. I can test this by clicking on the Detect button at the bottom and then selecting one of my images. It accurately detected each of the three cans in my image.
• Now I want to detect that in text. To do this I’ll add a gallery and for the gallery data source, I’ll point it to Object Detector. Watch my video included to see the exact code I use here. I can also add code to get a count of each type of can in the picture and filter it, so it only shows the objects in the image, not all the objects I have in the model.
• I’m also going to add a look-up command because my galleries are small; I typically do not add a look up inside a gallery because it will do a look up individually for every row which would not work well with large galleries.
• To do a look up command I’ll need a data source; I’ll use the Object Detection Examples data source which is where my inventory is stored. Again, watch my video to see this in more detail, but what it will do now is give me an inventory amount for each can in the image.
• You can also see in my video how I add a patch command to the save button the I’ve added in my gallery, so this will be sent to my database and add in whatever I see in the image to the inventory total I see in the text box. I’ll also show you what this app looks like in the real world by opening this Power App I built on my phone, taking a new photo of soda cans and testing the object detection out and it will automatically add the cans detected to my inventory. Very cool stuff!

My example shows how we can use the AI Builder for object detection in the real world. This inventory system I created will add things automatically to my inventory and reduce human error.

If you’d like to learn more about this awesome new feature in Power Apps or about Power Apps in general, we can help. Our On-Demand Learning platform has over 50 courses, including courses on Power Apps and the Power Platform. To get starting learning how to build your own app, our App in a Day course is a free course that is a perfect way to get started using Power Apps. Click below to subscribe to your FREE App in a Day course!