|
Augmented Reality |
Augmented Reality (AR) is a form of virtual reality that registers images of a virtual environment (VE) with the real world. One of the best overviews of the technology is Ron Azuma's A Survey of Augmented Reality (in a special issue of Presence on AR). A good starting place on the web is the Augmented Reality Homepage. There are several AR specific workshops as well as broader conferences that include AR topics that are producing excellent papers.
Mixed Reality (MR) is a related term with a broader definition by Paul Miligram and Fumio Kishino in their 1994 paper A Taxonomy of Mixed Reality Visual Display. Mixed reality refers to a multi-axis spectrum of technology that covers AR, VR, telepresence, and other related systems. Portability was not an axis used in the MR paper but perhaps it should be included. Wearable computing applications generally provide unregistered, text/graphics information using a monocular HMD. These systems are more of a "see-around" setup and not an Augmented Reality system by the narrow definition. The computing platforms and display devices used in AR are often developed for more general wearable applications.
Location-aware computing is a recent form of Mixed Reality. There are
commercial GPS enabled navigation systems for cars, planes that offer a
2d map. Some systems for hikers show elevation and terrain details. Cell
phones and PDAs are being augmented with location technology. They do not
utilize a 3D display, but they have a very large potential market. GeoVector
of San Francisco is one company with a broad (perhaps questionable) patent
base for these systems. Worldboard
is a project to attach web information to real world. Then as the objects
move about, they can be accessed with location aware or proximity devices.
The project concept specifies no hardware or access mechanisms and so could
be used for various forms of location aware or AR systems. Information
in Place is a commercial outgrowth of the Worldboard project.
Back to Contents
Four major classes of AR can be distinguished by their display type. Optical See-Through AR uses a transparent Head Mounted Display (HMD) to display the virtual environment (VE) directly over the real wold. Projector Based AR uses real world objects as the projection surface for the VE. Video See-Through AR uses an opaque HMD to display merged video of the VE with and view from cameras on the HMD. Monitor Based AR also uses merged video streams but the display is a more conventional desktop monitor or a hand held display. It is perhaps the least difficult AR setup, as it eliminates HMD issues.
Prime examples of an Optical See-through AR system are the various augmented medical systems. The MIT Image Guided Surgery has concentrated on brain surgery. UNC has been working with an AR enhanced ultrasound system and other ways to superimpose radiographic images on a patient. There are many other Optical See-through systems, as it seems to be the main direction for AR. Unfortunately there is a lack of suitable see-through HMDs. The Sony Glasstron was popular but the high-resolution version is nolonger made. A monocular HMD from Colorado Microdisplay was shown at Comdex this past month, but its commercial availability is unknown. Kaiser makes a see-through version of their Pro-View XL series. Another issue for Optical See-through AR is the alignment of the HMD optics with the real world. A good HMD allows adjustments to fit the eye position and comfort of individual users. It should also be easy to move it out of the way when not needed. However, these movements will alter the registration of the VE over the real world and require re-calibration of the system. An expensive solution would be to instrument the adjustments, so the system could automagically compensate for the motion. No such instrumented HMDs exist to my knowledge.
Projector based AR was the subject of a breakout session at the 2nd International Workshop on AR (IWAR 1999). It has applications in industrial assembly, product visualization, etc. Projector based AR is also well suited to multiple user situations. Alignment of projectors and the projection surfaces is critical for successful applications. Teleconferencing can also be considered a form of projector (or monitor) based AR, especially in some of the newer versions that utilize computer generated avatars for the remote participants.
Video See-Through AR was the subject of research at UNC using several custom HMDs. This approach is a bit more complex than optical see-through AR, requiring proper location of the cameras. However, video composition of the real and virtual worlds is much easier. There are a variety of solutions available including chroma-key and depth mapping. Mixed Reality Systems Lab (MRSL) of Japan presented a stereo video see-through HMD at ISAR 2000. This device addresses some of the parallax related to location of the cameras vs eyes.
Monitor Based AR is the most commercially mature form. Princeton Video Image, Inc. has developed a technique for merging graphics into real time video streams. Their work is regularly seen as the first down line in American football games. It is also used for placing advertising logos into various broadcasts.
Telerobotics and telepresence applications often fall into the Video
See-through AR or monitor based AR classification. The USC
Immersivion is an interesting new development that combines a 360 degree
video camera from Panoram
Technologies with an HMD. As the viewer turns their head, they see
a different portion of the video world. USC is planning to use the system
for psychological desentitization training and other research. They also
have plans to include computer overlays, making it AR by the narrow definition.
Back to Contents
The are three main technical issues for AR systems are tracking, calibration
and interaction between the real and virtual worlds.
Back to Contents
Location and orientation tracking is probably the hardest problem for AR. Many of the tracking solutions are technologies are shared with VR systems, however the requirements of AR are much more stringent. Drift, lag and other errors that may be tolerable in a VR setup quickly become problematic when attempting trying to register the virtual world images to the real world.
Indoor AR allows for a tightly controlled environment and the use of a wide range of tracking techniques. Inertial trackers such as the Intersense IS-300 and InterTrax2 provide some orientation and motion tracking, but are not accurate enough alone for AR work. Magnetic and many optical tracking solutions require preparation of the space with devices and targets. Vision based tracking has been a popular research topic for AR systems. Medical systems in particular can place fiducial targets in the camera view. Optical tracking has been a popular topic for research papers at recent conferences. See the reference list below for links to the conferences and sometimes the proceedings. The UNC Hi-Ball optical tracker, recently commercialized by 3rdTech, uses multiple IR sensors to track IR sources mounted on the ceiling. Large areas indoor areas can be setup for highly accurate tracking with this system.
The Shared Space project at the U.Washington HITLAB is an AR system that utilizes vision-based (fiducial) tracking for collaborative and other applications. Targets are places on square tiles and virtual images appear over the tiles in a see-through HMD as they are manipulated. The virtual objects have interaction behaviors that come into play when two or more tiles are brought together. This is a very interesting interaction technique. The project has made their source code free for non-commercial research as the AR Toolkit
Tracking outdoors introduces a number of new problems and possible solutions. Global Positioning Satellites (GPS, etc.) provide wide area location finding, but accuracy and availability are limited. While stationary GPS survey devices can provide centimeter or better accuracy, they are far from real time. A more realistic expectation for moving Differential GPS is 5 to 10 meters, on a good day. Many environmental artifacts, such as trees, mountains and buildings ("Urban Canyons"), can hinder GPS availability.
One effective approach for improving tracking results is the use of estimation and prediction. The combination of best estimate of location and orientation with a good prediction model can greatly reduce dynamic errors and compensate for system delays. Several researchers have utilized time series and kalman filters as reported in Ron Azuma's survey article.
The most promising tracking solutions for AR appear to be hybrid systems
that combine GPS, magnetic, inertial, and vision based tracking. The US
DARPA funded a project on warfighter visualization called GRIDS
(Geospatial Registration of Information for Dismounted Soldiers). USC,
UNC and HRL Laboratories LLC collaborated to develop a hybrid tracking
system for outdoor AR in unprepared areas.
Back to Contents
Calibration and registration of the world views is the next hardest problem for AR. The view parameters for rendering the virtual world must closely match the real world optics. Registration errors as small as one pixel can be detected in some circumstances. A See-Through AR system adds to the calibration problem with alignment of the HMD and eye. It is easier to produce registered images on a Monitor or Video See through systems. Optical tracking techniques are particularly useful on these systems, as the video is already being captured and processed. Calibration is often a very complex process requiring expert setup. A paper at ISAR 2000 from POSTECH (Korea) discussed a technique for autocalibration for video based AR. Calibration and registration are popular topics for research papers, so check through the past event websites and proceedings.
The interaction of the real and virtual worlds is a large issue for AR systems. This interaction includes the user interface (UI) as well as how the worlds are merged perceptually and in the VE simulation. AR User interfaces offer addition issues and some possible solutions for general 3D UI. Real devices and objects can be used as the controls instead of purely virtual ones. This offers additional haptic feedback, possibly improving control. Occlusion is a difficult issue for optical see-through and projector based AR. If the real world objects are known and closely tracked, they can obscure the virtual objects. It is another matter entirely to obscure real objects with virtual ones. Generally, the virtual objects are a see-through occlusion.
Kiyoshi Kiyokawa of Communications Research Laboratory, Japan is developing
an optical see-through HMD that supports occlusion of real world objects.
It uses a second LCD to block light from the real world, allowing virtual
objects to display without ghost effects. The HMD was demonstrated at Siggraph
2000 and a paper discussing it was presented at ISAR 2000.
Back to Contents
The portability of AR computers is another area of technical development. Wearable computers tend to be several generations behind desktop systems in processor and graphics support. Many commercial wearables use Intel Pentium processors with basic SVGA support. Alternatives such as the Transmeta Caruso (LINK) are providing competition. For those willing to forgo the x86 architecture, the ARM processors, particularly the Intel Strong ARM, is available in some wearable systems. This may preclude use of the Microsoft Windows family operating system, but such user interfaces are not well suited to wearable configurations. The new Nvidia GeForce 2 Go graphics chip promises to bring better 3D capabilities to notebooks and wearable computers. This should be a big help to AR developers. Meanwhile, fast systems capable of supporting advanced tracking and graphics generation require more processing power than available in wearable format. High end AR systems tend to use large backpacks or are not very portable setups.
One major barrier to commercial success of AR (and mobile computing
in general) is user resistance. Many people do not want to wear or carry
the computer system. The systems may be uncomfortable or perhaps unsightly.
Industrial workers have been subjected to jests and ridicule by coworkers,
severely impacting their motivation to use the equipment. These and other
human factors issues will need to be addressed by AR researchers and those
pursuing commercial endeavors.
Back to Contents
UNC - Azuma Augmented Reality
The definitive AR survey article from Presence 1997
http://www.cs.unc.edu/~azuma/azuma_AR.html
Taxonomy of Mixed Reality Visual Displays
1994 article defining Mixed Reality by Miligram and Kisnino
http://gypsy.rose.utoronto.ca/people/paul_dir/IEICE94/ieice.html
The Augmented Reality Homepage
AR Portal site sponsored by Rockwell Science Center
http://www.Augmented-Reality.org/
Jim Vallino's Augmented Reality Page
http://www.cs.rit.edu/~jrv/research/ar/
AR Forum e-mail list
Signup here for fairly active and informative e-mail list
http://ARforum.listbot.com/
Augmented Reality info/resources from Bauhaus
university Weimar
http://www.uni-weimar.de/~grether/
Wearable Computing and Augmented Reality resources
from U.Newfoundland
A good resource that may have been lost forever due to a server
crash
http://monet.engr.mun.ca/linkbuilder/list4989.html
Sony CSL's Augmented Reality links
projects, companies, events, etc.
http://www.csl.sony.co.jp/projects/ar/ref.html
Augmented Reality Links from the ARVIKA project
http://www.arvika.de/www/e/topic3/augmented.htm
See Event Calendar at Augmented Reality.org
Event list from ARVIKA Project
http://www.arvika.de/www/e/topic2/veranstaltungen.htm
ISMR 2001
International Symposium on Mixed Reality
March 14-15, 2001, Yokohama Japan
http://www.mr-system.co.jp/ismr/
1st International Symposium on Smart Graphics
March 21-23, 2001, Hawthorne, NY, USA
http://www.smartgraphics.org
ACM CHI 2001
March 31-April 5, 2001, Seattle WA, USA
http://www.acm.org/sigchi/chi2001
Eurographics Workshop on Virtual Environments
May 16-18, 2001, Stuttgart Germany
http://vr.iao.fhg.de/ipt-egve
ICAV3D
EUROIMAGE International Conference on Augmented,
Virtual Environments and Three-Dimensional Imaging
May 30 - June 1, 2001, Mykonos, Greece
http://www.iti.gr/ICAV3D/
HCI International
9th International conference on Human Computer
Interaction
August 5-10, 2001, New Orleans, LA, USA
http://hcii2001.engr.wisc.edu/
IMC 2000
Intelligent Interactive Assistance & Mobile
Multimedia Computing
http://www.rostock.igd.fhg.de/~imc2000/
IEEE IWAR: International Workshop on Augmented
Reality
1998, 1999
http://hci.rsc.rockwell.com/iwar/
Web Proceedings of IWAR'99
http://hci.rsc.rockwell.com/iwar/99/WebProceedings/
ISAR 2000
IEEE and ACM International Symposium on Augmented
Reality
http://www.Augmented-Reality.org/isar2000
http://wwwbruegge.informatik.tu-muenchen.de/projects/isar/2000/
ISWC 2000; International Symposium on Wearable
Computers
http://iswc.gatech.edu/
SPIE 1998: Telemanipulator and Telepresence
Technologies V (AM16)
http://www.spie.org/web/meetings/calls/pe98/isam/mras/am16.html
Artma Medical Technologies
computer-assisted surgical systems based on AR
http://www.artma.com/
GeoPerception
Personal Pilot for GIS-AR
http://www.geoperception.com/Products.htm
GeoVector
Mostly working on location aware cell phone apps, etc.
http://www.geovector.com/
Princeton Video Image
AR for live video - sports, ad placement, etc.
http://www.pvi-inc.com/
ATR- Ivan Poupyrev Augmented and Mixed Reality
http://www.mic.atr.co.jp/~poup/research/ar/
ARVIKA Consortium
AR for development, production and servicing.
Possibly the largest AR research effort.
Web site has a wealth of information, images, etc.
http://www.arvika.de/www/index.htm
Columbia University
Columbia: Mobile Computing Lab
http://www.cs.columbia.edu/mcl/
Columbia University: Virtual Worlds Research
http://www.cs.columbia.edu/graphics/projects/virtual-worlds.html
Columbia KARMA Wearable Computer Project
http://www.cs.columbia.edu/graphics/projects/karma/karma.html
Columbia: MARS - Mobile Augmented Reality
http://www.cs.columbia.edu/graphics/projects/mars/mars.html
DARPA GRIDS
Geospatial Registration of Information for Dismounted Soldiers
research by HRL Laboratories LLC, USC and UNC
http://deimos.usc.edu/~suyay/grids/grids-home.html
ECRC 1994/95 Augmented Reality Projects
http://www.ecrc.de/research/uiandv
Augmented Reality for Mechanical Maintenance and
Repair
http://www.ecrc.de/research/uiandv/gsp/ECRCToday/mechrep.html
ECRC GRASP: Applications
http://www.ecrc.de/research/uiandv/gsp/applications.html
EC JRC: 3D Reconstruction and Augmented Reality
- Home Page
http://mortimer.jrc.it/sba/3d.htm
INIRA Syntim Research Group
http://www-syntim.inria.fr/syntim
Augmented Reality as part of computer vision research
http://www-syntim.inria.fr/syntim/analyse/video-eng.html
MIT
MIT AI Lab HCI Project
http://www.ai.mit.edu/projects/hci/hci.html
MIT Medical Vision Group Home Page
http://www.ai.mit.edu/projects/medical-vision/
MIT: Image Guided Surgery
http://www.ai.mit.edu/projects/medical-vision/surgery/surgical_navigation.html
MIT: Medical Images
http://www.ai.mit.edu/people/leventon/Images/
MIT Augmented Reality Through Wearable Computing
http://testarne.www.media.mit.edu/people/testarne/TR397/main-tr397.html
Mixed Reality Systems Company of Japan
http://www.mr-system.co.jp/index_j.shtml
Nara IST- Yoshihiro Ban: Industrial Work Assisting
System Based on Augmented Reality
http://chihara.aist-nara.ac.jp/people/94/yosihi-b/research/AR-Web/index.html
Rockwell Science Center
http://hci.rsc.rockwell.com/
U.Central Florida
UCF's Internet 2 Augmented Reality Site
http://odalab.ucf.edu/internet2/
UCF: Augmented Reality for Radiographic Positioning
http://www.creol.ucf.edu/~rolland/bones.html
UDortmund: Co-Habited Mixed-Reality Information
Spaces
http://www-ai.cs.uni-dortmund.de/FORSCHUNG/PROJEKTE/COMRIS/index.eng.html
UHannover: EViTA
Enhanced Virtual Telepresence Animation
http://www.tnt.uni-hannover.de/plain/project/3dmod/evita/overview.html
U.North Carolina, Chapel Hill
UNC: Video see-through HMDs for AR
http://www.cs.unc.edu/~us/web/headmounts.htm
UNC Gallery of Augmented Reality HMDs
http://www.cs.unc.edu/~keller/lapro/Augmented_Reality_HMDs/gallery.html
UNC Medical Augmented Reality Project
http://www.cs.unc.edu/~us
Primary topic was medical AR but concentrated on many research issues.
Follow links on main page for info (and videos) on hybrid trackers, calibration
of magnetic trackers, latency manamgement, etc.
U.Nottingham: Augmented Reality for Subsurface
Data Visualisation
http://www.nottingham.ac.uk/iessg/isgres32.html
research spun into Augmented Reality Co UK
http://www.augmentedreality.co.uk
USC CGIT Augmented Reality
http://www.usc.edu/dept/CGIT/
USC: ImmersiVision | Panoramic Video
http://www-scf.usc.edu/~tpintari/immersivision/index.html
U.Surrey: Mechatronic Systems & Robotics
Research Group - Augmented Reality Research.
http://galileo.mech.surrey.ac.uk/Activities/AugReality/ar.html
USNavy BARS - Battlefield Augmented Reality
System
http://ait.nrl.navy.mil/vrlab/projects/BARS/BARS.html
UToronto ETC-Lab Augmented and Virtual Reality
Home Page
http://vered.rose.utoronto.ca/
UW HITLab
Research Page:
http://www.hitl.washington.edu/research
UW- HITLab Shared Space Project:
http://www.hitl.washington.edu/projects/shared_space
UW- Mark Billinghurst, Research Associate
http://www.hitl.washington.edu/people/grof
"Collaborative Mixed Reality" paper ISMR 1999
http://www.hitl.washington.edu/publications/r-98-36/
Sony Augmented Reality & Computer Augmented
Environments
http://www.csl.sony.co.jp/project/ar/ref.html
WorldBoard -- Augmented Reality
http://www.worldboard.com/
GMate YOPY
Linux-based PDA with HMD
http://www.gmate.co.kr
Genesis Technology Group, Inc
http://www.genesistech2000.com/product.html
IBM
wearable site search (IBM System Journal articles, prototype info,
etc.)
http://www.ibm.com/Search?v=9&q=wearable
Wearable Prototype 1998
http://www.ibm.com/News/ls/1998/09/jp_3.phtml
BBC News 1999: IBM Japan eyes wearable PC
http://news.bbc.co.uk/hi/english/sci/tech/newsid_538000/538072.stm
MIT Wearables
MIT: Wearable Computing Intro Page
http://lcs.www.media.mit.edu/projects/wearables/
MIT Wearables Links
http://lcs.www.media.mit.edu/projects/wearables/wearlinks.html
MIT Media Lab Diorama Project
http://diorama.www.media.mit.edu
Nijmegen U.: Pen-enabled Mobile and Wearable
Computers
http://hwr.nici.kun.nl/pen-computing
http://hwr.nici.kun.nl/pen-computing/pen-computing-wearables.html
Portable-Computer.com
http://www.portable-computer.com/
Sight Systems
ELVIS Wearable Computer for low vision enhancement
http://www.sightsystems.com/elvis.htm
TUDelft: Mobile Multimedia Communication (MMC)
1996-2000 Subprojects on work coordination, user interface (incl
HMDs),
http://www.mmc.tudelft.nl
User interface for MMC
http://www.mmc.tudelft.nl/subproject/userint/default.asp
Some research transfered to Ubiqutious Computing Project
http://www.ubicom.tudelft.nl
Virtual Vision Corporation
http://www.virtualvision.com/
ViA, Inc The Flexible PC Company
http://www.flexipc.com
WearCam.org, UTWCHI, and Steve Mann's Personal
Web Page/research
http://wearcam.org/
MIT: Wearables Central
http://wearables.ml.org/
Wearables Central
http://reg16.admin.rochester.edu/index.html
Wearables Web Ring's
http://profiles.yahoo.com/wearablesring
WearableGear.com
http://www.wearablegear.com/
Wireless Symposium/Portable by Design
http://www.WirelessPortable.com/west/
Xybernaut Corporation
http://www.xybernaut.com