VR News http://www.vrnews.com

March 2000 Technology Review

Alternative Interface Technologies

By Jerry Isdale

Standard VR interfaces are quite novel to main stream computing but there are quite a few technologies can be considered alternative. Vision enhancement systems, which combine head mounted displays (HMD) with minature cameras (visible, infra red, etc) are commercially available, but I will talk about those in November's Augmented Reality column. Exoskeleton systems are progressing in capability and quality, but I'll talk about them in October's Haptics column. This month we are going to look into two technologies that don’t fit any other planned topics. Eye Tracking and Neural Interfaces.

One of the premier research facilities working in this area is the US Air Force Research Lab's Human Effectiveness: Crew Systems Interface Division (AFRL/HEC). They put together a comprehensive description of alternative control devices in the "Nonconventional Controls" chapter of "Handbook of Human Factors and Ergonomics" (2^nd Ed, 1997.) A shorter, more focused version summarizing "Hands-Free Input Devices for Wearable Computers" (G.L. Calhoun and G.R. McMillan) was presented at "Human Interaction with Complex Systems (HICS) '98". (Proceedings available in the IEEE/CS digital library It provides good insights on how to combine inputs such as speech, gesture, electro-graphic and eye-based control technologies. For example, eye gaze can quickly designate an XY location and speech can trigger selection ("Delete That"). The AFRL/HEC team also contributed the chapter "Brain Actuated Controls and HMDs" to that excellent VR reference book "Head-Mounted Displays: Designing for the User" by James Melzer, Kirk Moffitt (McGraw Hill, ISBN 0070418195).

Eye Tracking has been widely used cognitive studies of attention and a number of commercial vendors provide solutions. There are several methods for tracking the eyes. Perhaps the simplest eye based control is a blink detector. It is fairly easy to detect whether the eyes are open or closed using a simple light source and photo-cell. However, winking only gives a binary state, temporarily blocks vision and may not be a very natural control.

Electrooculargraphy (EOG) measures the electrical potential of the skin around the eyes to detect the faint electrostatic field that rotates with the eye. These signals are created by corneal-retinal potential

(voltage difference between the metabolically active retina and relatively quiescent cornea.) as well as muscles that orient and focus the eye. Neural sensing techniques described later can be used detect these signals. However EOG is subject to drift and other variations and thu not well suited for direction of gaze tracking. It can be used effectively for basic four direction sensing (up, down, left, right).

Andrew Duchowski established the VR Eye Tracking (VRET) lab at Clemson University in 1998 as a complement to Clemsons existing VR lab. He worked with ISCAN and Virtual Research to integrate a VR8 HMD with a binocular video-based corneal reflection (single Purkinje image) eye tracker. The binocular eye tracker allows the measurement of vergence eye movements, thus providing the capability for calculating the 3d world coordinates of the user's gaze.

Andrew and an associate will be presenting a short course at SIGGRAPH 2000 entitled " Eye-Based Interaction in Graphical Systems: Theory & Practice". The course will provide information on how to setup and utilize this kind of equipment. Andrew is also organizing the "Eye Tracking Research & Applications Symposium 2000", November 6th-8th 2000, in Palm Beach Gardens, FL, USA.

One application of eye tracking plays on the physiology of the fovea of the eye, which has resolution higher than the periphery. Using eye gaze tracking, a system can determine the area of interest and improve resolution in this area. Steve Senger of University of Wisconsin - La Crosse, has been experimenting with eye-gaze directed computation for structure building and extraction from medical data sets.

The human body produces a myriad of signals that can be sensed by modern electronic equipment and interfaced to a computer. Basic body functions such as heart and respiratory rate, blood oxygen, blood pressure, temperature, etc are easily sampled with non-invasive technology. . Some signals are useful in determining a person's response to a situation. Others may be useful in controlling the computer. FitSense Inc and Lifeshirt.com are two of the many companies selling these products.

The electrical resistance of the skin (Galvanometric Skin Response or GSR) is one very easy signal to detect. Discovogue of Italy is marketing a GSR product called MindDrive that slips on the finger and drives a host of computer games. An SDK is available for MS Windows.

The most interesting body sensing systems use the complex electrical signals produced by muscle and neural activity. Electrooculargrapy (EOG), mentioned earlier is one such signal. Elecromyography (EMG) is the sensing of electrical signals from muscle contractions. Electroencephalographic (EEG) is the sensing of neural activity in the brain. Magnetoencephalograph (MEG) is a related technique for high accuracy localization of electromagnetic sources of brain activity using superconducting quantum interference devices (SQUID). EMG equipment is rather large and expensive but produces high fidelity brain activity maps.

Space precludes detailed background on EEG/EMG technology but there are excellent articles available on the web. Hugh Lusted and Ben Knapp of the now defunct BioControl Systems wrote an article for the October 1996 issue of Scientific America. Delsys Inc, a manufacturer of EMG systems, has a good tutorial on surface EMG (as opposed to invasive needles). Colorado State U. has some excellent background and links pages. They also make some EEG data sets and MathLab programs available for download. Jack Culpepper of Harvey Mudd College and Will Penny at Oxford University are two recent researchers working on sophisticated pattern recognition systems for brain computer interfaces. The EC Virtual Reality Environments for Psycho-neuro-physiological Assessment and Rehabilitation (VREPAR) project has also done some work in the area of VR and neuroscience. Their web site includes full PDF copies of the recent books "Virtual Environments in Clinical Psychology and Neuroscience",and "Virtual Reality in Neuro-psycho-physiology".

One device used in several experiments is the Cyberlink from Brain Actuated Technologies. Using 3 electrodes the system can be used for EEG, EMG and EOG. The provided software extracts 10 frequency "Brainfingers" derived from EEG and EMG signals that be used as separate continuos or discrete (thresholding) control channels. The ARFL team used a Cyberlink in a discrete control study which showed subjects’ reaction times to visual stimuli were found to be 15% faster with the Cyberlink EMG button than with a manual button. Last year, Wendy Amai led a team from Sandia National Labs used the Cyberlink's eye tracking ability to provide left/right signals to teleoperate a mobile robot.

Paras Kaul used the IBVA Inc three electrode system to control a VR and music generating environment.

Teresa Mann of Immersion Music got her PhD from MIT Media lab for the "Conductor’s Jacket", an biosensor based system to control and generate music from arm gestures from respiration, heart rate, temperature, skin conductance and EMG. Keith Lockhart conducted the Boston Pops with the Conductors Jacket in a 1998 event that included visual presentation of the electrical traces.

Biofeeedback researchers have found that control of EEG can train the mind back to healthy brain states. This "neurofeedback" is used for many conditions and disabilities such as Attention Deficit Hyperactivity Disorder (ADHD), specific learning disabilities, sleep problems in children, teeth grinding, chronic pain, mood disorders such as anxiety and depression, as well as for more severe conditions such as medically uncontrolled seizures, minor traumatic brain injury, or cerebral palsy.

Mark Swan created "Brain Wave Animation" using the Brain Master single/dual channel EEG device. The neurofeedback controls the speed of animation playback, along with more conventional graph and glyph displays. Further developments could provide a more virtual environment. David Kaiser of EEGSpectrum presented a paper at Medicine Meets VR 2000 giving documention for increased benefits of more engaging and immersive neurofeedback environments. His system used several 2d and 3d game environments from Neurocybernetics Inc. The 3d results were preliminary and more data is needed to firm up the conclusions. However, patients were more likely to agree to more sessions on the 3d environments. Other studies have shown that neurofeedback effectiveness improves with the number of sessions.

So what about output devices? Well there is Motionware, Virutal Motion's synthetic motion simulation using galvanic (electrical) stimulation of the vestibular system. We will update you on it in next month's Motion Platform column. Muscle stimulators are receiving a fair bit of interest and press lately. MIT Technology Review has a fascinating article in their March/April 2000 issue on the use of EEG control of Neurocontrol's Freehand functional electrical stimulation (FES) system. A quadriplegic patient with the FES implants uses a multi-electrode EEG skull cap to control his hand grasp and arm motion. Other work by the Cleveland FES Institute has used the implants to help paraplegics walk. These are still fairly rudimentary muscle control systems, but progress has been heartening to many. ElectroAccupuncture by Voll's Method (EAV) is another electrical stimulation therapy that reportedly detects and helps remove blockages in the body's regulation systems. Some of these clinical devices, such as the Accutron from Microcurrent Research Inc, include a serial connection for computer control and recording.

Dave Warner's recent PhD thesis "First Principles of Physio Informatic Ssystems" attempts to provide a conceptual framework which can be used as a guiding heuristic tool for the design and development of interactive interface systems for human computer interaction. The work Dave and his cohorts at Puslar.org and related institutes have done with exotic IO devices is quite amazing. His thesis puts forth a symbology to describe the transformation of information between the objective world, body physiology and the mind. However, it lacks a truly rigorous mathematical derivation. Such a solid foundation would provide true scientific background and lead the way to engineering practices for developing advanced human computer interface systems. Perhaps continuing work on Pulsar's various Biocentric and "Biotic"projects will move his work a bit in that direction. I suspect we will have to wait for someother enterprising and visionary doctoral candidate to take on this daunting task.