A wearable system for mobility improvement of visually impaired people

Degradation of the visual system can lead to a dramatic reduction of mobility by limiting a person to his sense of touch and hearing. This paper presents the development of an obstacle detection system for visually impaired people. While moving in his environment the user is alerted to close obstacles in range. The system we propose detects an obstacle surrounding the user by using a multi-sonar system and sending appropriate vibrotactile feedback. The system aims at increasing the mobility of visually impaired people by offering new sensing abilities.     

A dataset for the visually impaired walk on the road

As a special group, visually impaired people (VIP) find it difficult to access and use visual information in the same way as sighted individuals. In recent years, benefiting from the development of computer hardware and deep learning techniques, significant progress have been made in assisting VIP with visual perception. However, most existing datasets are annotated in single scenario and lack of sufficient annotations for diversity obstacles to meet the realistic needs of VIP. To address this issue, we propose a new dataset called Walk On The Road (WOTR), which has nearly 190 K objects, with approximately 13.6 objects per image. Specially, WOTR contains 15 categories of common obstacles and 5 categories of road judging objects, including multiple scenario of walking on sidewalks, tactile pavings, crossings, and other locations. Additionally, we offer a series of baselines by training several advanced object detectors on WOTR. Furthermore, we propose a simple but effective PC-YOLO to obtain excellent detection results on WOTR and PASCAL VOC datasets. The WOTR dataset is available at https://github.com/kxzr/WOTR     

Artificial intelligence for visually impaired

The eyes are an essential tool for human observation and perception of the world, helping people to perform their tasks. Visual impairment causes many inconveniences in the lives of visually impaired people. Therefore, it is necessary to focus on the needs of the visually impaired community. Researchers work from different angles to help visually impaired people live normal lives. The advent of the digital age has profoundly changed the lives of the visually impaired community, making life more convenient. Deep learning, as a promising technology, is also expected to improve the lives of visually impaired people. It is increasingly being used in the diagnosis of eye diseases and the development of visual aids. The earlier accurate diagnosis of the eye disease by the doctor, the sooner the patient can receive the appropriate treatment and the better chances of a cure. This paper summarises recent research on the development of artificial intelligence-based eye disease diagnosis and visual aids. The research is divided according to the purpose of the study into deep learning methods applied in diagnosing eye diseases and smart devices to help visually impaired people in their daily lives. Finally, a summary is given of the directions in which artificial intelligence may be able to assist the visually impaired in the future. In addition, this overview provides some knowledge about deep learning for beginners. We hope this paper will inspire future work on the subjects..     

Mixture reality-based assistive system for visually impaired people

Individuals with visual impairments often face challenges in their daily lives, particularly in terms of independent mobility. To address this issue, we present a mixed reality-based assistive system for visually impaired individuals, which comprises a Microsoft Hololens2 device and a website and utilizes a simultaneous localization and mapping (SLAM) algorithm to capture various large indoor scenes in real-time. This system incorporates remote multi-person assistance technology and navigation technology to aid visually impaired individuals. To evaluate the effectiveness of our system, we conducted an experiment in which several participants completed a large indoor scene maintenance task. Our experimental results demonstrate that the system is robust and can be utilized in a wide range of indoor environments. Additionally, the system enhances environmental perception and enables visually impaired individuals to navigate independently, thus facilitating successful task completion.     

Context-aware obstacle detection for navigation by visually impaired

This paper presents a context-aware smartphone-based based visual obstacle detection approach to aid visually impaired people in navigating indoor environments. The approach is based on processing two consecutive frames (images), computing optical flow, and tracking certain points to detect obstacles. The frame rate of the video stream is determined using a context-aware data fusion technique for the sensors on smartphones. Through an efficient and novel algorithm, a point dataset on each consecutive frames is designed and evaluated to check whether the points belong to an obstacle. In addition to determining the points based on the texture in each frame, our algorithm also considers the heading of user movement to find critical areas on the image plane. We validated the algorithm through experiments by comparing it against two comparable algorithms. The experiments were conducted in different indoor settings and the results based on precision, recall, accuracy, and f-measure were compared and analyzed. The results show that, in comparison to the other two widely used algorithms for this process, our algorithm is more precise. We also considered time-to-contact parameter for clustering the points and presented the improvement of the performance of clustering by using this parameter.     

Image-to-MIDI mapping based on dynamic fuzzy color segmentation for visually impaired people

In this paper, the RGB ratio is defined according to a reference color so that an image can be transformed from a conventional color space to the RGB ratio space. Different to traditional distance measurement, a road color model is determined by an ellipse area in the RGB ratio space enclosed by the estimated boundaries. The proposed dynamic fuzzy logic, where fuzzy membership functions are defined according to estimated boundaries, is introduced to implement clustering rules, such that each pixel will have its own fuzzy membership function corresponding to its intensity. A basic neural network is trained and used to achieve parameter optimization. Experimental results for road detection demonstrate the robustness of the proposed approach to variations in intensity. To provide obstacle information, especially for visually impaired people, Musical Instrument Digital Interface (MIDI) is introduced as the sound generator, and image-to-MIDI mapping algorithm is proposed. Experimental results show that the proposed method can adapt to various road types, and the resulting audio information successfully indicates the position and size of obstacles.     

Supporting visually impaired children with software agents in a multimodal learning environment

Visually impaired children have a great disadvantage in the modern society since their ability to use modern computer technology is limited due to inappropriate user interfaces. The aim of the work presented in this paper was to develop a multimodal software architecture and applications to support learning of visually impaired children. The software architecture is based on software agents, and has specific support for visual, auditory and haptic interaction. It has been used successfully with different groups of 7-8 year-old and 12 year-old visually impaired children. In this paper we discuss the enabling software technology and interaction techniques aimed to realize our goal and our experiences in the actual use of the system.     

Spatio-temporal analysis for obstacle detection in agricultural videos

Autonomous mobile vehicles are becoming commoner in outdoor scenarios for agricultural applications. They must be equipped with a robot navigation system for sensing, mapping, localization, path planning, and obstacle avoidance. In autonomous vehicles, safety becomes a major challenge where unexpected obstacles in the working area must be conveniently addressed. Of particular interest are, people or animals crossing in front of the vehicle or fixed/moving uncatalogued elements in specific positions. Detection of unexpected obstacles or elements on video sequences acquired with a machine vision system on-board a tractor moving in cornfields makes the main contribution to this research. We propose a new strategy for automatic video analysis to detect static/dynamic obstacles in agricultural environments via spatial-temporal analysis. At a first stage obstacles are detected by using spatial information based on spectral colour analysis and texture data. At a second stage temporal information is used to detect moving objects/obstacles at the scene, which is of particular interest in camouflaged elements within the environment. A main feature of our method is that it does not require any training process. Another feature of our approach consists in the spatial analysis to obtain an initial segmentation of interesting objects; afterwards, temporal information is used for discriminating between moving and static objects. To the best of our knowledge in the field of agricultural image analysis, classical approaches make use of either spatial or temporal information, but not both at the same time, making an important contribution. Our method shows favourable results when tested in different outdoor scenarios in agricultural environments, which are really complex, mainly due to the high variability in the illumination conditions, causing undesired effects such as shadows and alternating lighted and dark areas. Dynamic background, camera vibrations and static and dynamic objects are also factors complicating the situation. The results are comparable to those obtained with other state-of-art techniques reported in literature.     

Uncalibrated obstacle detection using normal flow

This paper addresses the problem of obstacle detection for mobile robots. The visual information provided by a single on-board camera is used as input. We assume that the robot is moving on a planar pavement, and any point lying outside this plane is treated as an obstacle. We address the problem of obstacle detection by exploiting the geometric arrangement between the robot, the camera, and the scene. During an initialization stage, we estimate an inverse perspective transformation that maps the image plane onto the horizontal plane. During normal operation, the normal flow is computed and inversely projected onto the horizontal plane. This simplifies the resultant flow pattern, and fast tests can be used to detect obstacles. A salient feature of our method is that only the normal flow information, or first order time-and-space image derivatives, is used, and thus we cope with the aperture problem. Another important issue is that, contrasting with other methods, the vehicle motion and intrinsic and extrinsic parameters of the camera need not be known or calibrated. Both translational and rotational motion can be dealt with. We present motion estimation results on synthetic and real-image data. A real-time version implemented on a mobile robot, is described.     

The interaction experiences of visually impaired people with assistive technology: A case study of smartphones

Globally, the number of visually impaired people is large and increasing. Many assistive technologies are being developed to help visually impaired people, because they still have difficulty accessing assistive technologies that have been developed from a technology-driven perspective. This study applied a user-centered perspective to get different and hopefully deeper understanding of the interaction experiences. More specifically, this study focused on identifying the unique interaction experiences of visually impaired people when they use a camera application on a smartphone. Twenty participants conducted usability testing using the retrospective think aloud technique. The unique interaction experiences of visually impaired people with the camera application, and relevant implications for designing assistive technologies were analyzed.Relevance to industryThe considerations for conducting usability testing and the results of this study are expected to contribute to the design and evaluation of new assistive technologies based on smartphones.     

Accurate 3D-vision-based obstacle detection for an autonomous train

In this paper we present a 3D-vision based obstacle detection system for an autonomously operating train in open terrain environments. The system produces dense depth data in real-time from a stereo camera system with a baseline of 1.4 m to fulfill accuracy requirements for reliable obstacle detection 80 m ahead. On an existing high speed stereo engine, several modifications have been applied to significantly improve the overall performance of the system. Hierarchical stereo matching and slanted correlation masks increased the quality of the depth data in a way that the obstacle detection rate increased from 89.4% to 97.75% while the false positive detection rate could be kept as low as 0.25%. The evaluation results have been obtained from extensive real-world test data. An additional stereo matching speed-up of factor 2.15 was achieved and the overall latency of obstacle detection is considerably faster than 300 ms.     

AudioBrowser: a mobile browsable information access for the visually impaired

Although a large amount of research has been conducted on building interfaces for the visually impaired that allows users to read web pages and generate and access information on computers, little development addresses two problems faced by the blind users. First, sighted users can rapidly browse and select information they find useful, and second, sighted users can make much useful information portable through the recent proliferation of personal digital assistants (PDAs). These possibilities are not currently available for blind users. This paper describes an interface that has been built on a standard PDA and allows its user to browse the information stored on it through a combination of screen touches coupled with auditory feedback. The system also supports the storage and management of personal information so that addresses, music, directions, and other supportive information can be readily created and then accessed anytime and anywhere by the PDA user. The paper describes the system along with the related design choices and design rationale. A user study is also reported.     

Hierarchical adaptive stereo matching algorithm for obstacle detection with dynamic programming

An adaptive weighted stereo matching algorithm with multilevel and bidirectional dynamic programming based on ground control points （GCPs） is presented. To decrease time complexity without losing matching precision, using a multilevel search scheme, the coarse matching is processed in typical disparity space image, while the fine matching is processed in disparity-offset space image. In the upper level, GCPs are obtained by enhanced volumetric iterative algorithm enforcing the mutual constraint and the threshold constraint. Under the supervision of the highly reliable GCPs, bidirectional dynamic programming framework is employed to solve the inconsistency in the optimization path. In the lower level, to reduce running time, disparity-offset space is proposed to efficiently achieve the dense disparity image. In addition, an adaptive dual support-weight strategy is presented to aggregate matching cost, which considers photometric and geometric information. Further, post-processing algorithm can ameliorate disparity results in areas with depth discontinuities and related by occlusions using dual threshold algorithm, where missing stereo information is substituted from surrounding regions. To demonstrate the effectiveness of the algorithm, we present the two groups of experimental results for four widely used standard stereo data sets, including discussion on performance and comparison with other methods, which show that the algorithm has not only a fast speed, but also significantly improves the efficiency of holistic optimization.     

A sonar approach to obstacle detection for a vision-based autonomous wheelchair

An advanced prototype Computer Controlled Power Wheelchair Navigation System or CCPWNS has been developed to provide autonomy for highly disabled users, whose mix of disabilities makes it difficult or impossible to control their own power chairs in their homes. The working paradigm is “teach and repeat” a mode of control for typical industrial holonomic robots. Ultrasound sensors, which during subsequent autonomous tracking will be used to detect obstacles, also are active during teaching. Based upon post-processed data collected during this teaching event, elaborate trajectories–which may involve multiple direction changes, pivoting and so on, depending upon the requirements of the typically restricted spaces within which the chair must operate–will later be called upon by the disabled rider. An off-line postprocessor assigns an ultrasound profile to the sequence of poses of any taught trajectory. Use of this profile during tracking obviates most of the inherent problems of using ultrasound to avoid obstacles while retaining the ability to near solid objects, such as when passing through a narrow doorway, where required by the environment and trajectory objectives. The work in this article describes a procedure to obtain consistent maps of sonar boundaries during the teaching process, and a preliminary approach to use this information during the tracking phase. The approach is illustrated by results obtained by using the CCPWNS prototype.     

Real-time pedestrian crossing lights detection algorithm for the visually impaired

In defect of intelligent assistant approaches, the visually impaired feel hard to cross the roads in urban environments. Aiming to tackle the problem, a real-time Pedestrian Crossing Lights (PCL) detection algorithm for the visually impaired is proposed in this paper. Different from previous works which utilize analytic image processing to detect the PCL in ideal scenarios, the proposed algorithm detects PCL using machine learning scheme in the challenging scenarios, where PCL have arbitrary sizes and locations in acquired image and suffer from the shake and movement of camera. In order to achieve the robustness and efficiency in those scenarios, the detection algorithm is designed to include three procedures: candidate extraction, candidate recognition and temporal-spatial analysis. A public dataset of PCL, which includes manually labeled ground truth data, is established for tuning parameters, training samples and evaluating the performance. The algorithm is implemented on a portable PC with color camera. The experiments carried out in various practical scenarios prove that the precision and recall of detection are both close to 100%, meanwhile the frame rate is up to 21 frames per second (FPS).     

Mobile device accessibility for the visually impaired: problems mapping and recommendations

Universal Access in the Information Society - Mobile devices can be an important ally that improves the quality of life of visually impaired people by permitting greater independence in the...     

一种障碍物模型检测算法

针对观测场景结构复杂、建模困难的问题,提出了一种基于运动目标跟踪的场景障碍物模型检测方法.结合对目标的检测与跟踪过程,利用空间投影关系确定场景中障碍物的位置,并将其映射到场景模型上,即可得到场景观测模型.该方法只需要对目标运动进行简单地跟踪和统计,不需要进行三维结构恢复以及图像拼接处理,能够有效解决场景的在线自适应建模问题.     

Understanding the requirements of geographical data for blind and partially sighted people to make journeys more independently

Previous research has highlighted that blind and partially sighted people find various factors inhibit their abilities to make journeys. This paper proposes that the lack of accurate, appropriate and usable geographical data is one of the reasons for this and these can be tracked back to core human factors issues such as situational awareness, mental workload and environmental ergonomics. Following a review of applicable literature a hierarchical task analysis was performed to better understand the problems in terms of the complexity of various journey types and to identify the geographical data requirements in order to make successful journeys. The task analysis produced a number of results including highlighting four underlying principles which have an impact on the data requirements during any given journey. Finally the need for accessible and accurate geographical data requirements is introduced as a result of the literature review and the task analysis. These highlight the information required in order to facilitate more accessible travel for blind and partially sighted people by providing geographical information about their surroundings in a relevant, meaningful and usable way.     

Walking strategies of visually impaired people on trapezoidal- and sinusoidal-section tactile groundsurface indicators

The visual system in walking serves to perceive feedback or feed-forward signals. Therefore, visually impaired persons (VIP) have biased motor control mechanisms. The use of leading indicators (LIs) and long canes helps toimprove their walking efficiency. The aims of this study were to compare the walking efficiency of VIP on trapezoidal- and sinusoidal-section LIs using an optoelectronic motion analysis system. VIP displayed a significantly longer stance phase, a shorter swing phase and shorter step and stride lengths when they walked on the sinusoidal LIthan when they walked on the trapezoidal LI. Compared with the trapezoidal LI, VIP walking on the sinusoidalLIdisplayed significantly lower joint ranges of motion. The centre of mass lateral displacement was wider for VIP walking on the sinusoidal LI than on the trapezoidal LI. Some significant differences were also found in sighted persons walking on both LIs. In conclusion, the trapezoidal shape enabled visually impaired subjects to walk more efficiently, whereas the sinusoidal shape caused dynamic balance problems.

Statement of Relevance: These findings suggest that VIP can walk more efficiently, with a lower risk of falls, on trapezoidal-section than on sinusoidal-section LIs. These results should be considered when choosing the most appropriate ground tactile surface indicators for widespread use.     

An enhanced text detection technique for the visually impaired to read text

An enhanced text detection technique (ETDT) is proposed, which is expected to aid the visually impaired to overcome their reading challenges. This work enhances the edge-preserving maximally stable extremal regions (eMSER) algorithm using the pyramid histogram of oriented gradients (PHOG). Histogram of oriented gradients (HOG) derived from different pyramid levels is important while detecting maximally stable extremal regions (MSER) in the ETDT approach because it gives more spatial information when compared to HOG information from a single level. To group text, a four-line, text-grouping method is newly designed for this work. Also, a new text feature, Shapeness Score is proposed, which significantly identifies text regions when combined with the other features based on morphology and stroke widths. Using the feature vector of dimension 10, the J48 decision tree and AdaBoost machine learning algorithms identify the text regions in the images. The algorithm yields better results than the existing benchmark algorithms for the ICDAR 2011 born-digital dataset and must be improved with respect to the scene text dataset.