Roy Choudhury's student advances mobile augmented reality technology

Anyone who has seen a spy movie remembers their favorite agent donning special eyewear or using a device to scan surrounding objects and extract information about them. Once an action movie fantasy, augmented reality technology is now almost, well, reality.

“You are interacting with the real world, and putting virtual information on top of that,” said Puneet Jain, a PhD student being advised by Professor Romit Roy Choudhury.

Jain has been working on the technology for about two years. He is conducting research and developing the software along with Roy Choudhury and Justin Manweiler of IBM Research in New York. The NSF, Nokia, and Google are funding the project.

The technology, called OverLay, works inside a smartphone and essentially combines the camera and the sensor data to make sense of the surroundings in real time. Once OverLay knows the user is looking at an object, say a pair of shoes on display, relevant information about those shoes can pop up immediately on the screen. The technology will be a breakthrough for marketers, who will be able to advertise products at all times.

Such forms of mobile augmented reality have been a tantalizing topic for researchers since the dawn of smartphones. Jain's passion for the project stems from prior experience in developing technology that identifies the location of objects outdoors. Although GPS could be used outside, geometric understanding of the environment isn't accurate enough in indoor environments.

Roy Choudhury noted "combining vision and sensing achieves the desired outcome. Specifically, as users look at different objects in their environment, the sensor data is used as geometric constraints.” 

For instance, OverLay observes that objects A and B are statistically separated by certain angles, objects B and C are typically separated by 10 seconds in time, and so on.

“With these observations,” Jain said, “we came up with an optimization framework to build a geometry out of our surroundings, and once we built that layout, we were able to drastically speed-up computer vision.”

The uses for the technology extend beyond defense and surveillance, as seen loosely in fictional spy movies. Jain believes it could be beneficial for infrastructure management, museums, shopping and privacy.

For instance, an out-of-reach object, such as a broken air conditioner, could be labeled with specific information about what part needs to be fixed, using one’s phone from a distance. Through augmented reality, museums could provide more interactivity and education about artifacts and exhibits. If photography is prohibited for certain paintings, the camera could automatically become inactive when it’s pointed at these exhibits.

In the future, the technology could be deployed by the likes of Google and Amazon, but infrastructural, financial, and legal obstacles must be overcome before a public release of that scale. For the time being, the researchers are working closely with the Illinois Distributed Museum to iron out issues and make mobile augmented reality accessible for a wider audience.

Jain and his colleagues presented their work at HotMobile, the International Workshop on Mobile Computing Systems and Applications, winning Best Demo. They published a full paper detailing their work at ACM MobiSys 2015, an international conference in Florence, Italy.