The Geek Weekly: Innovate

The New World of Augmented Reality

From the Editors of The Geek Weekly

Virtual reality has become a little less, well, virtual.

The new growth field in computer research is called “augmented reality” (AR), which mixes up real-world experiences with those that are computer-generated.

The best-known example of augmented reality that you can see on a weekly basis is the yellow “first down” line that shows up on TV broadcasts of American football games. The game is really being played on a football field, but that yellow line isn't something players or fans in the stands can see. The experience has been “augmented” for TV viewers by the addition of a computer-generated graphic.

In just a few years, however, AR will likely enhance a variety of our senses -- from sight to sound to smell and touch. It's also likely to be mobile. By deploying a simple pair of glasses augmented with information graphics and audio, your experience of walking down a street will be enhanced with data about restaurants, landmarks or the quickest route to your destination. Researchers are currently working on a wide variety of AR applications that can be used for anything, from offering help with military training to augmenting surgical procedures to helping tourists navigate new surroundings.

“We're talking about augmenting one's view or experience of the world and doing it in real time,” says Steven Feiner, a professor of computer science at Columbia University and director of the Computer Graphics and User Interfaces Laboratory. Feiner is one of the foremost AR researchers in the world.

Although virtual reality (VR) attempts to replace the real world with a virtual world, AR instead adds the virtual world to the real world, so the real world is still here to see, hear, touch, smell and taste. “It's blurring the line between what is real and what is not real,” says Greg Davis, general manager of Total Immersion, a French company that markets augmented reality products to entertainment companies and museums.

Small Attractions and Head-mounted Displays
Right now, most AR applications involve the enhancement of your real experience with video imagery, and perhaps, audio. In addition to the first-down line, AR is currently used to enhance your TV viewing experiences with other spectator sports. For example, Sport Vision, a Chicago-based firm, has created an AR application featuring details about a car/driver that appear on screen during a NASCAR race.

Six Flags launched a ride last summer based on the blockbuster Batman film The Dark Knight. As a part of the attraction, you look into the mirror but see the face of one of the Joker's clown-masked henchman virtually imposed over your own image.

Video games are also ripe for AR. Sony's Eye of Judgment game for PlayStation 3, which came out in late 2007, involves battling with playing cards. The game uses the PlayStation Eye's camera peripheral to identify cards using a barcode-like system and to provide you with information about your hand, including possible attacks and special abilities.

How AR Works
Ivan Sutherland, considered the father of computer graphics, developed the first AR system in the 1960s. It consisted of a head-mounted display system with little monitors that optically combined your view of the real world with the virtual material you saw on the screens, according to Feiner. It wasn't until the 1970s that groups started expanding on this concept using see-through displays and developing applications, such as those providing aircraft pilots information while in flight. In the 1990s, Feiner built a device that combined AR with the trend of mobile computing, developing a mobile augmented reality system (MARS) that put augmented images and experiences in wearable, head-mounted mobile devices.

The chief components of an AR system are hardware, software, some sort of display to see the augmented view combined with the real-life view, and often a camera to capture surroundings. If the system is mobile, a GPS system and wireless network would help to track your physical location and orientation of your head and communicate with the computer to pull up virtual information about that location and what you're seeing.

The Future of AR
The tools for AR are being built into some of the very devices you use everyday. Cell phones now often feature digital video cameras. Computer makers are also consulting with AR researchers about what types of features they need to build into the phone's processor to enable AR, such as graphics or media processors, Feiner says.

Using available tools, such as GPS and a smartphone, a team of engineers at the Nokia Research Center in Helsinki, Finland, recently unveiled a prototype system that could help travelers navigate a new city. The technology-enhanced phone was able to pinpoint the location of objects the camera was directed toward. The phone retrieved information about landmarks from an external database, also signaling to the user that they could download more details about a landmark or a restaurant from the Web.

Researchers are also developing medical applications so that surgeons who operate on a patient by watching a computer monitor will now be able to look right at the patient and have information and diagrams overlay their view to, for example, ensure they are making incisions in the proper locations. Military apps might allow troops to receive critical information about their surroundings.

While visual applications of AR abound, and audio apps run a close second, researchers are also exploring whether they can include other sensory materials to augment a user's sense of smell, touch or even taste. Some of these applications, however, may prove a bit more challenging for the consumer market.

For example, projects have been done wafting certain smells under the nose of an AR user, but they involve attaching various tubes containing different smells to pipe the odors to the nose. “The problem is that you might want to have a wide range of different things to smell and you would have to support that by having lots of aerosol cans hooked up,” Feiner says. “Once you're given the smell, you also have the problem of trying to get rid of it very quickly.”

Even without the added effects of aromas, AR is poised to enter the mainstream. “People are less questioning of how it works and seem ready to enjoy the experience,” Davis says.

Elizabeth Wasserman is a freelance writer and editor based in Fairfax, Va. She writes for a variety of publications, including Congressional Quarterly and Inc. magazine, and she edits the online publication CIO Strategy Center.