In the last two years, I have conducted many experiments with Augmented Reality (or AR) to explore and demonstrate the value that Augmented Reality brings to many industries – from recreating historic sites and future buildings to air quality and flight path visualizations.
Before Augmented Reality, information was restricted to 2D screens or paper. Augmented Reality changes all this freeing data to be displayed in context of the 3D world of reality for the first time.
While early days this is game changing, reducing the cognitive load for many different tasks. From locating, visualizing, and managing both indoor and outdoor infrastructure, to visualizing new projects that have yet to be constructed, to visualizing the past. Augmented Reality promises to increase understanding and productivity across industries. I am excited to share just some of the work that I have done with FME and its ability to drive Augmented Reality workflows.
The GPS chips inside mobile phones are good enough for many scenarios. It is always possible to adjust the model location on the screen. While many AR scenarios do not need a precise AR model placement, for some use cases, accuracy is absolutely critical.
This is why we conducted an experiment. We used our AR app, FME AR, with an Arrow Gold, a high-accuracy GNSS receiver. The device was kindly loaned to us from Eos Positioning Systems for two weeks of testing. Our findings show that it is possible to achieve a very high level of accuracy in AR model placement and data collection. This blog post will explain how and why!
The problem
It is very hard to find a phone without a built-in GPS module these days. However, the GPS chips inside the phones and tablets may not be very accurate. They serve well for car navigation, for following trails in the forest or searching geocaches – an error of 5 to 10 meters does not significantly affect these activities. In case of geocaching, an error has certain health benefits – people spend more time outside looking for hidden treasures.
The nature of work conducted by many of our customers requires a much better accuracy than regular or even the most expensive phones can provide. This is especially true when the professionals deal with hidden objects such as underground infrastructure. This is also true when professionals work in a dense city environment with multiple similar assets in close proximity to each other such as manhole covers. It becomes very hard or nearly impossible to correctly place an AR model so that it fits into the real world. As a result, the low accuracy becomes a significant factor. This hinders the adoption of AR technology for these high accuracy solutions.
The compass is another phone sensor that is important for geospatially-aware AR. It is also not the most accurate sensor. The device orientation error can reach 10 and even 15 degrees. Adjusting rotation might not be easy if we move away from the model origin. Resultantly, it is better to orient the model as soon as it is loaded.
The solution
GNSS receivers are well known for their much higher accuracy than a simple GPS device. Moreover, by connecting a receiver to an RTK network, it is possible to achieve centimeter accuracy.
In theory, we knew a GNSS device with RTK should give us the required accuracy. However, it didn’t feel right to just tell our customers to use such a device without having a hands-on experience.
This is why I contacted Eos Positioning Systems.
I asked whether we could rent a unit for a couple of weeks. The company kindly offered an Arrow Gold unit free of charge. In just a few days, we began our testing.
First tests
Connecting our Eos Arrow Gold receiver to the iOS devices (iPhone and iPad) was as simple as connecting a set of earbuds. The receiver works over Bluetooth and just appears among available devices. It was a bit harder to connect to an Android device (we had to turn on a developer mode there). Nonetheless, it didn’t take more than a couple of minutes.
A special app, EosToolsPro (App Store, Google Play), turns a phone or a tablet into a controller. From there, we can see all the necessary details on the screen.
Our first test was very simple. We wanted to load a model with known coordinates and see it at those coordinates. No more than a few centimetres away rather than three, five, or ten meters off.
For testing model placement, we chose the most reliable dataset, Geodetic Control Monuments. The dataset contains geodetic control markers. This is a network of interconnected markers in the ground which have accurately determined coordinates. In practical terms, using this dataset was a guarantee that if our model loaded away from a marker, it was the problem on our side, not the data.
After some struggles with creating a correct dataset (yes, sometimes I make mistakes when I create workspaces), we went on a field trip and started testing. At this moment, we understood that we were facing another problem, but this was a very good kind of problem.
The difference between the marker location and the model location was so small. This was true to the point that the size of the iPad became a significant factor in model placement!
Where does the iPad “think” its centre is when loading an AR scene? Is it the centre of the device? Or is it the camera? Or maybe the GPS antenna inside? I don’t know the answer yet, but as you can see on the photos below, we had a pretty good match even with no knowledge about the centre.
The red and white pole is the AR model loaded at the marker location.
Compass Sensor Accuracy
At this point, we were still facing a problem of low accuracy compass sensors. Loading a small model that represents a point is not affected by it. However, larger models, if rotated, won’t look correctly. They would still need a manual adjustment. How can we do it using our high accuracy GNSS receiver?
I came up with the following idea: measure coordinates of two points and use them as parameters in the workspace that creates the model. Using these two sets of coordinates, we can create an augmented line. This will become a part of the whole AR model. Once the model is loaded, we, while standing on the first point, would see the line. Then, we will orient it toward the second point. This technique works really well. This way, the last part of the accurate model placement was resolved.
From now on, we know for sure. We are happy to share this information with everyone who needs a high-accuracy AR model placement. It really works!
After completing these steps, we began testing more interesting applications.
Scenario Testing
We see a great value in using AR for finding and inspecting the objects of the underground infrastructure. Examples include pipes and cables, and related items such as manholes, electrical boxes, light poles etc.
We tested and can confirm an accurate model placement in the scenario described earlier in this blog. The only change I made to the service was adding one more parameter. This is a point for replacing the compass sensor.
Note how the orientation technique works here:
Our tests were happening during an unprecedented amount of rain. Every time we had a break, we spent it outside doing more testing. Some tests look more compelling during the rain.
We all know that leaves during the fall can cover and even clog the manhole drains. Finding them might be challenging. With FME, it is very easy to create a service that will show all the drainage manholes around the submitted location. However, it might be hard to find them anyway if a phone GPS is not accurate enough. With Arrow Gold, we could see the markers directly over the manholes. This allows us to find and clean them right away.
Open datasets are published by many cities. This makes it really easy to create such a service not only for city workers, but also for any other organization. Once I created this service for Port Moody, the city where I live, I replicated the service for a few more municipalities around Metro Vancouver. This took about 30 minutes per city from start (googling “Open data Municipality Name”) to finish (testing whether the service works).
Data Collection
We also had a non-AR scenario. We wanted to try the data collection capabilities of another mobile app, FME Data Express. FME DX for short (App Store, Google Play). This mobile app runs FME Server workspaces from mobile devices. FME DX allows using different device sensors (i.e. GPS or camera) and technologies (i.e. QR codes) as input parameters.
More details about data collection with FME DX in general can be found in this video.
We wanted to confirm two ideas with FME DX. First, that an external GPS works with FME DX, and second, the data we collect is really accurate.
I created a simple workspace that allows collecting points, lines, and polygons, and saves them into Google Sheets. This is a demo spreadsheet available for writing to anyone through a data collection server app.
In the following video, I collect a curb in the center of a cul-de-sac. I repeated the test several times with Arrow Gold connected and without it.
The following image shows the results of my collection. It is hard to see, but there are two green lines representing Arrow Gold. This shows that the difference between the two collection sessions is minimal.
This confirms that FME DX can be used with external GNSS devices for data collection.
Conclusion
During the two weeks of testing, we gathered enough data to confidently conclude that FME AR and FME DX work with external GNSS devices. It is possible to achieve a very high level of accuracy in AR model placement and data collection.
Using devices like Eos Arrow Gold complements a powerful combination of the FME platform products. FME Data Express collects highly accurate coordinates and sends them to FME Cloud. This is where a web service creates an AR model for that particular location and sends it back to the device. After, the FME AR app places the model at its true geographic position.
We would like to thank Eos Positioning Systems for letting us use the Arrow Gold GNSS receiver. Having this unit was crucial for understanding the possibilities of AR and data collection apps. Thanks to Steve Lawrence, Sales Manager at Eos, who helped us with setting up the device and resolving all the issues we had. We also would like to thank Metro Vancouver for giving us access to the local RTK network.
Let’s Talk About Augmented Reality (AR)
Wide adoption of AR is still some time away, but people who start now will be better positioned to be ahead of others when the time comes.
If you are as interested and excited as the team at Safe Software and I are about the possibilities that AR and data have to offer, give us a follow to watch first hand as I test out AR scenarios with FME:
Looking to get into more details or have questions about potential uses for AR and your organization? Or are you interested in sharing your experiences with AR and data? I would love to hear from you.
Reach out to me at dmitri.bagh@safe.com or to Safe Software at info@safe.com for an opportunity to connect and discover what AR could have in store for you.
Try FME for free and learn more about what FME and AR can do in our ‘Exploring Outdoor Infrastructure with AR’ webinar that is coming up soon!
Dmitri Bagh
Dmitri is the scenario creation expert at Safe Software, which means he spends his days playing with FME and testing what amazing things it can do.