Raphael Thys
At the start of May last year Microsoft announced a new feature for the augmented reality headset, HoloLens 2. Prior to the upgrade the headset was experiencing issues of moving environments confusing the headset’s sensors.
The issue at hand saw the augmented reality headset sensors conflicting with each other due to one tracking movement via camera and one via sensors. When used inside a car the headset was sensing movement but seeing a static environment around it causing ‘confusion’.
The new enhancement has been developed in collaboration with Volkswagen and is designed to present drivers with virtual objects inside and outside of the vehicle as well as showcasing a virtual map onto the dashboard of the car. HoloLens 2 can also show navigation arrows at key intersections and alert drivers of upcoming pedestrians who are crossing in front of the vehicle.
We spoke to Marc Pollefeys, director, Microsoft Mixed Reality and AI Lab Zurich, to find out more about this new technology.
Just Auto (JA): How did the partnership with Volkswagen come to be?
Marc Pollefeys (MP): In 2015, Volkswagen embarked on a research project using self-piloted vehicles and augmented reality to teach driving on a racing circuit.
Volkswagen hoped to use HoloLens for the project, but found that when the device was put into a moving vehicle, its sensors lost tracking and the holograms it normally displays disappeared.
Volkswagen researchers approached Microsoft for help and connected with myself, Microsoft director of science and an expert in 3D computer vision and machine learning.
The Volkswagen team travelled to Microsoft’s headquarters in Redmond, Washington to show myself and team its work with HoloLens and its vision for augmented reality technology. The two teams began collaborating around 2018 to develop the moving platform feature for HoloLens 2.
When in use within a vehicle how does the augmented reality work?
To address the issue of the sensors losing tracking, my team developed an algorithm that models the discrepancies between the sensors and allows HoloLens to continue tracking.
The new “moving platform” mode for HoloLens 2 overcomes a major limitation of mixed reality headsets and creates potential for the technology to be used in new ways — training drivers to handle challenging road conditions.
Could you explain the virtual map technology, which is presented on the dashboard of the car, and what this includes.
This is a demo application exploring how future navigation systems could look like that leverages moving platform ability. Note that it is not necessarily representative of plans for future technology or currently intended for broad use.
What are the key benefits of this technology for vehicle occupants?
In the future, augmented reality tools like moving platform mode will help enable seamless and connected mobility experiences, including receiving information, navigational assistance, and entertainment as people travel through their day.
Are there plans for future development of the technology/partnership with other companies which you are able to talk about?
The technology is already attracting interest from maritime companies and organizations, which have been using HoloLens to remotely connect maritime workers with mechanical experts through Microsoft Dynamics 365 Remote Assist.
Microsoft plans to further refine moving platform mode for use in elevators, trains, cars and other moving environments. There are multiple ways the technology could be used to connect the inside of vehicles with the outside world, such as helping truck or bus drivers navigate through narrow streets, identifying points of interest along driving routes or providing entertainment for passengers.