Application of AR for smart governance

Why am I writing this blog?

So that the words like “smart-governance” don’t seem like a far fetched dream. We have the tools to create the future, that once eluded us. I sincerely hope that after reading this you may get inspired and create something wonderful!

Introduction to smart government

It is about using technology to facilitate and support better planning and decision making. It is about improving democratic processes and transforming the ways that public services are delivered. It is a new way of governance relying on information and communication technologies and it is citizen centric, data driven and performance focused.

  • https://www.igi-global.com/dictionary/smart-government/45119
  • Obviously we can’t change all aspects of governance at once, so we have to start from somewhere. Let’s look at one particular public service and how it be delivered in a better manner.

    Okay, before you go ahead a little heads up. Rest of the blog is divided into two parts

    Part 1 : Solution to a problem area

    Part 2 : Development of that solution

    Part 1

    Introduction to the problem area

    Let’s take an example of James, who wants to open a restaurant. Currently the process of getting licenses and permits is a tricky one. Some of the broad steps James needs to take to get licenses are

    1. Go to the government’s website and register his business
    2. Apply for relevant licenses
    3. Get a time slot for inspector’s visit
    4. Inspector visits the site
    5. Inspector suggest changes for compliance.
    6. Again! the Inspector visits 😦
    7. Inspector approves

    In terms of efficiency, step number 4, 5, and 6 are the slowest which involve movement of a person from one place to another. But in today’s world, this is not needed since information can be transferred much more easily.

    Let’s simplify the inspection process a little bit more. Assume that James is only applying for a fire safety license. For such a license the inspection process takes two things in account.

    1. Number of fire extinguishers
    2. Placement of those fire extinguishers

    How did we solve it?

    We can ask the business owner, James to download an inspection AR app. Let’s see how the new experience looks like.

    1. James : Opens the app and points to a fire extinguisher.
    2. App : Recognizes the fire extinguisher and increase the counter by one.
    3. James : Measures the distance of the fire extinguisher from the exit by making a line.
    4. App : Records the distance and compares with the threshold value. If the distance falls within the limit, it gives the green signal.
    5. Steps 1 to 4 get repeated for any other fire extinguisher.

    Here is a video for you to understand this flow properly.

    Ideally, we can introduce few steps to make the flow much more robust like

    Step 0 – Attaching a QR code on the fire extinguishers when they are first installed

    Step 2.1 – Ask James to scan the QR code so that the meta data about the fire extinguisher gets updated. This is addressed in problem 2 of the loop holes section.

    Step 2.2 – James uploads a photo of the gauge and the top part. Which gets sent to the inspector to verify.

    Step 6 – Using the site plan of the restaurant we can show a view of where all the fire extinguishers are placed, are they meeting the current guidelines? And where are they supposed to be placed?

    Benefits

    1. Time Saved : As multiple trips of the inspector are not needed both James and the inspector save time.
    2. Reduced workload : The inspector will be only be required in cases where the restaurant owner is not addressing the flagged potential violations by the app.
    3. More Autonomy : Since inspector is not involved, James can perform the inspection any time.
    4. Less compliance violations : Since the app can flag the potential violations in real time, this will give the restaurant owner to rectify those issues and get a clearance immediately.

    Loop Holes

    No technology or idea is perfect at first. Same goes for this one. Here I have identified the problems with the proposed flow and respective potential solutions.

    Problem 1 : Recognizing the varying shapes and sizes of the fire extinguisher.

    Potential Solution : Typically a machine learning algorithm should be able to achieve a fairly high level of confidence in classifying objects as “Fire extinguishers” or “Not Fire Extinguishers”.

    Problem 2 : Getting meta data about fire extinguishers like size, date of expiry, type etc.

    Potential Solution : A scanner that scans the bar code and QR which should incorporate such data. Already solution like that exist in the market. Link

    Problem 3 : Measuring the distances when the fire extinguisher and the exit gate are on different planes separated by multiple planes.

    Potential Solution : Assuming that we can get highly accurate GPS co-ordinate measurement through a smartphone, we can ask the user to stand the two end points and drop pins. The pins can then be used to calculate the shortest distance between the ends. Or, following the similar mechanism as described in the flow, the user can drag lines on top of multiple planes which connect the two ends.

    What’s next?

    The idea of this prototype was not to present a full proof solution but to give a hint that new exciting technology like AR can be applied in the conventional world of governance. If you are interested about this topic then feel free to reach out. I will be happy to collaborate. In fact in the next section I have explained how I managed to develop this small prototype. Hope it helps!

    Part 2

    How to develop such an app?

    This app essentially uses three pieces of technology

    1. Image/Object Recognition
    2. Plane Recognition and Distance Measurement

    Image/Object Recognition

    Basically the app has to recognize the fire extinguisher. And how can we do that? Creating a recognition algorithm is beyond my capability. Luckily, we don’t have to do that. Unity3D provides AR modules that can handle the image recognition part.

    Steps Overview

    1. Get package “AR Foundation 2.2.0”
    2. Go to “AR session origin” gameobject.
    3. Add “AR tracked image manager”
    4. Create a “Reference Image Library” and assign the image you want to track as a texture.
    5. Add the created library as a reference library.
    6. Add a custom script “imageRecognitionScript”. Here is the link to the script.

    Here is the video which really helped in doing this step. Pretty much everything is taken from there.

    Plane Recognition and Distance Measurement

    Okay! now that our app has the ability to recognize the fire extinguisher (or at least the image of it), it should check whether the placement is correct or not. One way is to ask James (restaurant owner) to measure the distance manually and then enter it. We are not recommending that since it takes more effort and the data entered can’t be verified.

    We already have AR distance measurement technology (check the brilliant “Measure” app on iphone), why not use it.

    Plane Detection Steps Overview

    1. Add AR session gameobject from the XR package
    2. Create an AR session origin gameobject
    3. Add the AR plane manager script
    4. Create a plane prefab “AR plane visualizer”, the details of which you can see in the video.

    AR Distance Measurement Steps Overview

    1. Create a custom “Measurement Controller” script.
    2. Create a line renderer component which will act as the measurement tape.
    3. Instantiate the measurement tape’s end points at origin in the awake method. Make them inactive.
    4. Now in the Update(), write a block of code that gets the first touched point on the screen, starts a ray from that point to the world. Get the position and rotation of the point where the ray hits a plane. Place the first point of your line renderer.
    5. Similarly from the end point of the drag event, start a ray. Place the end point of the line renderer wherever the line hits the plane.
    6. Now that you have two points, you can easily calculate the length of the line using Distance() function.

    And that’s it, you have a functional app which recognizes the image/object and allows the user to measure the distance through AR. Magic! isn’t it?

    Team

    This idea was made possible by these awesome! folks – Abhay, Neeraj, Karan, Narendra and Ujjwal.

    Bibliography

    1. Smart Government – https://www.igi-global.com/dictionary/smart-government/45119
    2. AR Foundation with Unity3d and Creating An Augmented Reality Measuring Tape Tool – https://www.youtube.com/watch?v=vZalV7–_uA
    3. https://imectechnologies.com/fire-extinguisher-barcode-inspection-software-landing-r2/

    Leave a Reply

    Fill in your details below or click an icon to log in:

    WordPress.com Logo

    You are commenting using your WordPress.com account. Log Out /  Change )

    Twitter picture

    You are commenting using your Twitter account. Log Out /  Change )

    Facebook photo

    You are commenting using your Facebook account. Log Out /  Change )

    Connecting to %s