A distributed device diagnostics system utilizing augmented reality and 3D audio

Augmented reality (AR), combining virtual environments with the perception of the real world, can be used to provide instructions for routine maintenance and error diagnostics of technical devices. The Rockwell Science Center (RSC) is developing a system that utilizes AR techniques to provide “X-ray vision” into real objects. The system can overlay 3D rendered objects, animations, and text annotations onto the video image of a known object. An automated speech recognition system allows the user to query the status of device components. The response is given as an animated rendition of a CAD model and/or as auditory cues using 3D audio. This diagnostics system also allows the user to leave spoken annotations attached to device modules as ASCII text. The position of the user/camera relative to the device is tracked by a computer-vision-based tracking system using fiducial markers. The system is implemented on a distributed network of PCs, utilizing standard commercial off-the-shelf components (COTS).