Gaze perception and awareness in smart devices

Eye contact and gaze awareness play a significant role for conveying emotions and intentions during face-to-face conversation. Humans can perceive each other's gaze quite naturally and accurately. However, the gaze awareness/perception are ambiguous during video teleconferencing performed by computer-based devices (such as laptops, tablet, and smart-phones). The reasons for this ambiguity are the (i) camera position relative to the screen and (ii) 2D rendition of 3D human face i.e., the 2D screen is unable to deliver an accurate gaze during video teleconferencing. To solve this problem, researchers have proposed different hardware setups with complex software algorithms. The most recent solution for accurate gaze perception employs 3D interfaces, such as 3D screens and 3D face-masks. However, today commonly used video teleconferencing devices are smart devices with 2D screens. Therefore, there is a need to improve gaze awareness/perception in these smart devices. In this work, we have revisited the question; how to improve a remote user's gaze awareness among his/her collaborators. Our hypothesis is that ‘an accurate gaze perception can be achieved by the ‘3D embodiment’ of a remote user's head gesture during video teleconferencing’. We have prototyped an embodied telepresence system (ETS) for the 3D embodiment of a remote user's head. Our ETS is based on a 3-DOF neck robot with a mounted smart device (tablet PC). The electromechanical platform in combination with a smart device is a novel setup that is used for studying gaze awareness/perception in 2D screen-based smart devices during video teleconferencing. Two important gaze-related issues are considered in this work; namely (i) ‘Mona-Lisa Gaze Effect’ – the gaze is always directed at the person independent of his position in the room, and (ii) ‘Gaze Awareness/Faithfulness’ – the ability to perceive an accurate spatial relationship between the observing person and the object by an actor. Our results confirm that the 3D embodiment of a remote user head not only mitigates the Mona Lisa gaze effect but also supports three levels of gaze faithfulness, hence, accurately projecting the human gaze in distant space.     

Virtual Volumetric Graphics on Commodity Displays Using 3D Viewer Tracking

Three dimensional (3D) displays typically rely on stereo disparity, requiring specialized hardware to be worn or embedded in the display. We present a novel 3D graphics display system for volumetric scene visualization using only standard 2D display hardware and a pair of calibrated web cameras. Our computer vision-based system requires no worn or other special hardware. Rather than producing the depth illusion through disparity, we deliver a full volumetric 3D visualization—enabling users to interactively explore 3D scenes by varying their viewing position and angle according to the tracked 3D position of their face and eyes. We incorporate a novel wand-based calibration that allows the cameras to be placed at arbitrary positions and orientations relative to the display. The resulting system operates at real-time speeds (～25 fps) with low latency (120–225 ms) delivering a compelling natural user interface and immersive experience for 3D viewing. In addition to objective evaluation of display stability and responsiveness, we report on user trials comparing users’ timings on a spatial orientation task.     

基于标记点检测的视线跟踪注视点估计

传统的头戴式视线跟踪系统需要借助额外的头部位置跟踪器或其他辅助设备才能定位视线方向.针对该问题,提出一种基于标记点检测的注视点估计方法.该方法通过计算机视觉的方法检测标记点,建立场景图像与真实场景中计算机屏幕之间的空间关系,将场景图像中的注视点坐标映射到计算机屏幕中.实验结果表明,该方法简单易行,可以较好地估计出用户在...     

An auxiliary gaze point estimation method based on facial normal

Considering the main disadvantage of the existing gaze point estimation methods which restrict user’s head movement and have potential injury on eyes, we propose a gaze point estimation method based on facial normal and binocular vision. Firstly, we calibrate stereo cameras to determine the extrinsic and intrinsic parameters of the cameras; Secondly, face is quickly detected by Viola–Jones framework and the center position of the two irises can be located based on integro-differential operators; The two nostrils and mouth are detected based on the saturation difference and their 2D coordinates can be calculated; Thirdly, the 3D coordinates of these five points are obtained by stereo matching and 3D reconstruction; After that, a plane fitting algorithm based on least squares is adopted to get the approximate facial plane, then, the normal via the midpoint of the two pupils can be figured out; Finally, the point-of-gaze can be obtained by getting the intersection point of the facial normal and the computer screen. Experimental results confirm the accuracy and robustness of the proposed method.     

Fuzzy-based estimation of continuous Z-distances and discrete directions of home appliances for NIR camera-based gaze tracking system

With the development of eye gaze tracking technology, much research has been performed to adopt this technology for interfacing with home appliances by severely disabled and wheelchair-bound users. For this purpose, two cameras are usually required, one for calculating the gaze position of the user, and the other for detecting and recognizing the home appliance. In order to accurately calculate the gaze position on the home appliance that the user looks at, the Z-distance and direction of the home appliance from the user should be correctly measured. Therefore, stereo cameras or depth-measuring devices such as Kinect are necessary, but they have limitations such as the need for additional camera calibration, and low acquisition speed for two cameras or a large-size of Kinect device. To overcome this problem, we propose a new method for estimating the continuous Z-distances and discrete directions of home appliances using one (small-sized) near-infrared (NIR) web camera and a fuzzy system. Experimental results show that the proposed method can accurately estimate the Z-distances and directions to home appliances.     

Preferred position of visual displays relative to the eyes: a field study of visual strain and individual differences

At office workplaces equipped with visual display units (VDU) that were adjustable to various positions relative to the eyes short and long viewing distances from the eyes to the screen were imposed (mean value of about 63 and 92 cm) at two levels of screen height so that the visual target was either at eye level or 18 cm below, on the average. The change from far to near viewing distance produced a larger increase in eyestrain when the VDUs were at eye level. High screens resulted in greater eyestrain than low screens, as shown by correlations over subjects. When operators were free to adjust the most comfortable screen position, the group of 22 participants preferred viewing distances between 60 and 100 cm and vertical inclination of gaze direction between horizontal and - 16° downwards. However, within most subjects the range of preferred screen positions was much smaller. Between 3 days during a 1-month period the test-retest correlations of the preferred screen positions were highly significant, both for viewing distance and vertical gaze inclination. When operators were forced to work at a shorter distance than their preferred viewing distance they reported more visual strain. Thus, operators appear to prefer an individual adjustment of the screen relative to the eyes in order to avoid visual strain and discomfort at VDU work.     

A neural-based remote eye gaze tracker under natural head motion

A novel approach to view-based eye gaze tracking for human computer interface (HCI) is presented. The proposed method combines different techniques to address the problems of head motion, illumination and usability in the framework of low cost applications. Feature detection and tracking algorithms have been designed to obtain an automatic setup and strengthen the robustness to light conditions. An extensive analysis of neural solutions has been performed to deal with the non-linearity associated with gaze mapping under free-head conditions. No specific hardware, such as infrared illumination or high-resolution cameras, is needed, rather a simple commercial webcam working in visible light spectrum suffices. The system is able to classify the gaze direction of the user over a 15-zone graphical interface, with a success rate of 95% and a global accuracy of around 2 degrees , comparable with the vast majority of existing remote gaze trackers.     

Gaze directed displays as an enabling technology for attention aware systems

Visual information can in principle be dynamically optimised by monitoring the user’s state of attention, e.g. by tracking eye movements. Gaze directed displays are therefore an important enabling technology for attention aware systems. We present a state-of-the-art review of both (1) techniques to register the direction of gaze and (2) display techniques that can be used to optimally adjust visual information presentation to the capabilities of the human visual system and the momentary direction of viewing. We focus particularly on evaluation studies that were performed to assess the added value of these displays. We identify promising application areas and directions for further research.     

Driver’s eye-based gaze tracking system by one-point calibration

The accuracies of driver’s gaze detection by previous researches are affected by the various sitting positions and heights of drivers in case that initial calibration of driver is not performed. By using dual cameras, the driver’s calibration can be omitted, but processing time with complexity is increased. In addition, the problem of disappearing corneal specular reflection (SR) in the eye image as the driver severely turns his/her head has not been dealt in previous researches. To consider these issues, we propose a gaze tracking method based on driver’s one-point calibration using both corneal SR and medial canthus (MC) based on maximum entropy criterion. An experiment with collected data from 26 subjects (wearing nothing, glasses, sunglasses, hat, or taking various hand pose) in a vehicle, showed that the accuracy of the proposed method is higher than that of other gaze tracking methods. In addition, we showed the effectiveness of our method in the real driving environment.

     

Estimating the camera direction of a geotagged image using reference images

Millions of smart phones and GPS-equipped digital cameras sold each year, as well as photo-sharing websites such as Picasa and Panoramio have enabled personal photos to be associated with geographic information. It has been shown by recent research results that the additional global positioning system (GPS) information helps visual recognition for geotagged photos by providing valuable location context. However, the current GPS data only identifies the camera location, leaving the camera viewing direction uncertain within the possible scope of 360°. To produce more precise photo location information, i.e. the viewing direction for geotagged photos, we utilize both Google Street View and Google Earth satellite images. Our proposed system is two-pronged: (1) visual matching between a user photo and any available street views in the vicinity can determine the viewing direction, and (2) near-orthogonal view matching between a user photo taken on the ground and the overhead satellite view at the user geo-location can compute the viewing direction when only the satellite view is available. Experimental results have shown the effectiveness of the proposed framework.     

Effects of 3D perspective on head gaze estimation with a multiview autostereoscopic display

Head gaze, or the orientation of the head, is a very important attentional cue in face to face conversation. Some subtleties of the gaze can be lost in common teleconferencing systems, because a single perspective warps spatial characteristics. A recent random hole display is a potentially interesting display for group conversation, as it allows multiple stereo viewers in arbitrary locations, without the restriction of conventional autostereoscopic displays on viewing positions. We represented a remote person as an avatar on a random hole display. We evaluated this system by measuring the ability of multiple observers with different horizontal and vertical viewing angles to accurately and simultaneously judge which targets the avatar is gazing at. We compared three perspective conditions: a conventional 2D view, a monoscopic perspective-correct view, and a stereoscopic perspective-correct views. In the latter two conditions, the random hole display shows three and six views simultaneously. Although the random hole display does not provide high quality view, because it has to distribute display pixels among multiple viewers, the different views are easily distinguished. Results suggest the combined presence of perspective-correct and stereoscopic cues significantly improved the effectiveness with which observers were able to assess the avatar׳s head gaze direction. This motivates the need for stereo in future multiview displays.     

面向异构网的呼叫控制体系研究

促进多网融合是下一代网络工作的主要研究方向,如何把握切换时机,在异构网络中进行准确切换是保证用户获取最佳的通信质量的核心.提出了基于无线局域网即有资源以及用户所出位置决定切换策略的机制,对用户的移动进行预测,能够掌握较为准确的切换时机,在两个不同模式的网络之间进行切换.在此机制下,可以根据用户个性化的数据,计算获得根据新呼叫失败概率以及呼叫切换失败概率数据,从而为切换提供理论数据.通过仿真实验表明了新切换机制的有效性.     

Intelligent process control via gaze detection technology

Gaze detection is to locate the position on a monitor screen where a user is looking. In our work, we implement it with a computer vision system setting a camera above a monitor, and a user moves (rotates and/or translates) his face to gaze at a different position on the monitor. In our case, the user is requested to keep the pupils of his/her eyes fixed when he/she gazes at a different position on the monitor screen; though we are working on to relax this restriction. Till now, we have tried different methods and among them, the proposed Two Neural Networks method shows the best results. For application of the gaze detection techniques to the user-interface of a control program, we supplement two additional techniques. One is to drag a mouse cursor by moving his/her face after initially placing the cursor by gazing, and another is to click the cursor by winking one eye. These are applied to user-interface of a process control program in three different ways. Firstly, even when both hands are busy in doing a work, the user can still control the monitor screen by gazing, dragging and winking. Secondly, if the user is absent and not in front of the monitor when important states occur, then the system can record and replay them later when the user returns. Finally, as the system knows where the user is looking, if an emergency occurs at a different point from where he/she is looking, then the system can display a warning signal. This paper also includes subjective and objective experimental results obtained from the tests by applying the underlined techniques to a process control for chemical vapor decomposition.     

A light phenomenon perceived when viewing CRT-based VDT screens with positive image

The present article describes a light phenomenon seen during work with video display terminals with positive image. The phenomenon is perceived for a very short moment as a brightly shining horizontal line when the gaze moves downwards on the screen. The vertical gaze velocity as measured by electro-oculography was 7 to 12 m/s. The vertical velocity of the electron beam on two different terminal screens recorded by photosensitive diodes was 10 m/s. The screens were also rotated 90 degrees , 180 degrees and 270 degrees from normal position. The phenomenon then occurred only when the electron beam line and the gaze moved in the same direction. It is suggested that the light phenomenon is perceived during a short moment when the electron beam line and the gaze move in the same direction with about equal velocity.     

Energy optimization for mobile video streaming via an aggregate model

Wireless video streaming on smartphones drains a significantly large fraction of battery energy, which is primarily consumed by wireless network interfaces for downloading unused data and repeatedly switching radio interface. In this paper, we propose an energy-efficient download scheduling algorithm for video streaming based on an aggregate model that utilizes user’s video viewing history to predict user behavior when watching a new video, thereby minimizing wasted energy when streaming over wireless network interfaces. The aggregate model is constructed by a personal retention model with users’ personal viewing history and the audience retention on crowd-sourced viewing history, which can accurately predict the user behavior of watching videos by balancing “user interest” and “video attractiveness”. We evaluate different users streaming multiple videos in various wireless environments and the results illustrate that the aggregate model can help reduce energy waste by 20 % on average. In addition, we also discuss implementation details and extensions, such as dynamically updating personal retention, balancing audience and personal retention, categorizing videos for accurate model.     

基于PLC的供电倒闸技术实训系统的研制

为了强化变电运行人员倒闸作业技术水平,降低供电倒闸作业的错误率,在现有供电模拟盘的基础上,融合触摸屏技术、PLC控制技术、红外遥控技术,设计、开发一基于PLC的供电倒闸技术实训系统。系统能根据用户的操作,通过灯光、音响的进行动态反应,对用户的操作进行判断和提示,给出参考信息,提高倒闸操作的技术水平。同时该系统还采用红外遥控集中控制、方便用户使用。此外,系统也可用于相关职业教育中的供电技术教学,开展供电倒闸技术训练。     

A wide view auto‐stereoscopic 3D display with an eye‐tracking system for enhanced horizontal viewing position and viewing distance

This paper describes the development of auto‐stereoscopic three‐dimensional (3D) display with an eye‐tracking system for not only the X‐axis (right–left) and Y‐axis (up–down) plane directions but also the Z‐axis (forward–backward) direction. In the past, the eye‐tracking 3D system for the XY‐axes plane directions that we had developed had a narrow 3D viewing space in the Z‐axis direction because of occurrence of 3D crosstalk variation on screen. The 3D crosstalk variation on screen was occurred when the viewer's eye position moved back and forth along the Z‐axis direction. The reason was that the liquid crystal (LC) barrier pitch was fixed and the LC barrier was able to control the only barrier aperture position. To solve this problem, we developed the LC barrier that is able to control the barrier pitch as well as the barrier aperture position in real time, corresponding to the viewer's eye position. As a result, the 3D viewing space has achieved to expand up to 320–1016 mm from the display surface in the Z‐axis direction and within a range of ±267 mm in the X‐axis direction. In terms of the Y‐axis direction, the viewing space is not necessary to be considered, because of a stripe‐shaped parallax barrier.     

The Effect of Collaboration Styles and View Independence on Video-Mediated Remote Collaboration

This paper investigates how different collaboration styles and view independence affect remote collaboration. Our remote collaboration system shares a live video of a local user’s real-world task space with a remote user. The remote user can have an independent view or a dependent view of a shared real-world object manipulation task and can draw virtual annotations onto the real-world objects as a visual communication cue. With the system, we investigated two different collaboration styles; (1) remote expert collaboration where a remote user has the solution and gives instructions to a local partner and (2) mutual collaboration where neither user has a solution but both remote and local users share ideas and discuss ways to solve the real-world task. In the user study, the remote expert collaboration showed a number of benefits over the mutual collaboration. With the remote expert collaboration, participants had better communication from the remote user to the local user, more aligned focus between participants, and the remote participants’ feeling of enjoyment and togetherness. However, the benefits were not always apparent at the local participants’ end, especially with measures of enjoyment and togetherness. The independent view also had several benefits over the dependent view, such as allowing remote participants to freely navigate around the workspace while having a wider fully zoomed-out view. The benefits of the independent view were more prominent in the mutual collaboration than in the remote expert collaboration, especially in enabling the remote participants to see the workspace.     

Animatronic shader lamps avatars

Applications such as telepresence and training involve the display of real or synthetic humans to multiple viewers. When attempting to render the humans with conventional displays, non-verbal cues such as head pose, gaze direction, body posture, and facial expression are difficult to convey correctly to all viewers. In addition, a framed image of a human conveys only a limited physical sense of presence—primarily through the display’s location. While progress continues on articulated robots that mimic humans, the focus has been on the motion and behavior of the robots rather than on their appearance. We introduce a new approach for robotic avatars of real people: the use of cameras and projectors to capture and map both the dynamic motion and the appearance of a real person onto a humanoid animatronic model. We call these devices animatronic Shader Lamps Avatars (SLA). We present a proof-of-concept prototype comprised of a camera, a tracking system, a digital projector, and a life-sized styrofoam head mounted on a pan-tilt unit. The system captures imagery of a moving, talking user and maps the appearance and motion onto the animatronic SLA, delivering a dynamic, real-time representation of the user to multiple viewers.     

Finding User’s Interest Blocks using Significant Implicit Evidence for Web Browsing on Small Screen Devices

Recent researches on improving the efficiency and user experience of Web browsing on handhelds are seeking to solve the problem by re-authoring Web pages or making adaptations and recommendations according to user preference. Their basis is a good understanding of the relationship between user behaviors and user preference. We propose a practical method to find user’s interest blocks by machine learning using the combination of significant implicit evidences, which is extracted from four aspects of user behaviors: display time, viewing information items, scrolling and link selection. We also develop a customized Web browser for small screen devices to collect user behaviors accurately. For evaluation, we conduct an on-line user study and make statistical analysis based on the dataset, which shows that most types of the suggested implicit evidences are significant, and viewing information items is the least indicative aspect of user behaviors. The dataset is then processed off-line to find user’s interest blocks using the proposed method. Experimental results demonstrate the effectiveness of finding user’s interest blocks by machine learning using the combination of significant implicit evidences. Further analysis reveals the great effect of users and moderate effect of Websites on the usefulness of significant implicit evidences.