Local context-based recognition of sketched diagrams

We present a new methodology aimed at the design and implementation of a framework for sketch recognition enabling the recognition and interpretation of diagrams. The diagrams may contain different types of sketched graphic elements such as symbols, connectors, and text. Once symbols are distinguished from connectors and identified, the recognition proceeds by identifying the local context of each symbol. This is seen as the symbol interface exposed to the rest of the diagram and includes predefined attachment areas on each symbol. The definition of simple constraints on the local context of each symbol allows to greatly simplify the definition of the visual grammar, which is used only for further refinement and interpretation of the set of acceptable diagrams. We demonstrate the potential of the methodology using flowcharts and binary trees as examples.     

A model-based recognition engine for sketched diagrams

Many of today's recognition approaches for hand-drawn sketches are feature-based, which is conceptually similar to the recognition of hand-written text. While very suitable for the latter (and more tasks, e.g., for entering gestures as commands), such approaches do not easily allow for clustering and segmentation of strokes, which is crucial to their recognition. This results in applications which do not feel natural but impose artificial restrictions on the user regarding how sketches and single components (shapes) are to be drawn.This paper proposes a concept and architecture for a generic geometry-based recognizer. It is designed for the mentioned issue of clustering and segmentation. All strokes are fed into independent preprocessors called transformers that process and abstract the strokes. The result of the transformers is stored in models. Each model is responsible for a certain type of primitive, e.g., a line or an arc. The advantage of models is that different interpretations of a stroke exist in parallel, and there is no need to rate or sort these interpretations. The recognition of a component in the drawing is then decomposed into the recognition of its primitives that can be directly queried for in the models. Finally, the identified primitives are assembled to the complete component. This process is directed by an automatically computed search plan, which exhibits shape characteristics in order to ensure an efficient recognition.In several case studies the applicability and generality of the proposed approach is shown, as very different types of components can be recognized. Furthermore, the proposed approach is part of a complete system for sketch understanding. This system not only recognizes single components, but can also understand sketched diagrams as a whole, and can resolve ambiguities by syntactical and semantical analysis. A user study was conducted to obtain recognition rates and runtime data of our recognizer.     

Using data mining for digital ink recognition: Dividing text and shapes in sketched diagrams

The low accuracy rates of text-shape dividers for digital ink diagrams are hindering their use in real world applications. While recognition of handwriting is well advanced and there have been many recognition approaches proposed for hand drawn sketches, there has been less attention on the division of text and drawing ink. Feature based recognition is a common approach for text-shape division. However, the choice of features and algorithms are critical to the success of the recognition. We propose the use of data mining techniques to build more accurate text-shape dividers. A comparative study is used to systematically identify the algorithms best suited for the specific problem. We have generated dividers using data mining with diagrams from three domains and a comprehensive ink feature library. The extensive evaluation on diagrams from six different domains has shown that our resulting dividers, using LADTree and LogitBoost, are significantly more accurate than three existing dividers.     

A syntactic approach based on distortion-tolerant Adjacency Grammars and a spatial-directed parser to interpret sketched diagrams

This paper presents a syntactic approach based on Adjacency Grammars (AG) for sketch diagram modeling and understanding. Diagrams are a combination of graphical symbols arranged according to a set of spatial rules defined by a visual language. AG describe visual shapes by productions defined in terms of terminal and non-terminal symbols (graphical primitives and subshapes), and a set functions describing the spatial arrangements between symbols. Our approach to sketch diagram understanding provides three main contributions. First, since AG are linear grammars, there is a need to define shapes and relations inherently bidimensional using a sequential formalism. Second, our parsing approach uses an indexing structure based on a spatial tessellation. This serves to reduce the search space when finding candidates to produce a valid reduction. This allows order-free parsing of 2D visual sentences while keeping combinatorial explosion in check. Third, working with sketches requires a distortion model to cope with the natural variations of hand drawn strokes. To this end we extended the basic grammar with a distortion measure modeled on the allowable variation on spatial constraints associated with grammar productions. Finally, the paper reports on an experimental framework an interactive system for sketch analysis. User tests performed on two real scenarios show that our approach is usable in interactive settings.     

Online hand-sketched figure recognition

An online algorithm capable of recognizing hand-sketched symbols such as those used in flowcharts is presented. The algorithm requires no indication of symbol segmentations and no restrictions on the stroke sequence of symbols. The algorithm has three steps: (1) candidate figure extraction for each symbol based on a graph search and distance calculation between candidate figures and reference patterns, (2) selection of the symbol sequence which minimizes the total sum of these distances, (3) connection rules application.A recognition test performed on 100 hand-sketched flowcharts and block diagrams produced a recognition rate of 96.1%.     

Generating systems from multiple sketched models

Diagrams are often used to model complex systems: in many cases several different types of diagrams are used to model different aspects of the system. These diagrams, perhaps from multiple stakeholders of different specialties, must be combined to achieve a full abstract representation of the system. Many CAD tools offer multi-diagram integration; however, sketch-based diagramming tools are yet to tackle this difficult integration problem. We extend the diagram sketching tool InkKit to combine software engineering sketches of different types. Our extensions support software design processes by providing a sketch-based approach that allows the iterative creation of multiple outputs interacting with one another from the inter-linked models. We demonstrate that InkKit can generate a functional system consisting of a user interface with processes to submit and retrieve data from a database from sketched user interfaces designs and sketched entity relationship diagrams.     

基于ICA的在线掌纹识别

个人身份认证和鉴别在现在社会显示着重要的作用,作为一种准确而可靠的个人鉴定方式,生物识别已经引起了广泛的注意.掌纹作为一种相对较新的生物识别技术也有着独特的优点.而掌纹特征的提取和选择是整个识别中最关键的一个环节,主要利用ICA（独立主成分分析）方法对掌纹进行特征提取,实验证明,相比较PCA（主成分分析）方法,基于ICA方法具有更高的识别效率.     

Online recognition of handwritten music symbols

In this paper, we propose an effective online method to recognize handwritten music symbols. Based on the fact that most music symbols can be regarded as combinations of several basic strokes, the proposed method first classifies all the strokes comprising an input symbol and then recognizes the symbol based on the results of stroke classification. For stroke classification, we propose to use three types of features, which are the size information, the histogram of directional movement angles, and the histogram of undirected movement angles. When combining classified strokes into a music symbol, we utilize their sizes and spatial relation together with their combination. The proposed method is evaluated using two datasets including HOMUS, one of the largest music symbol datasets. As a result, it achieves a significant improvements of about 10% in recognition rates compared to the state-of-the-art method for the datasets. This shows the superiority of the proposed method in online handwritten music symbol recognition.     

Symbol recognition in electrical diagrams using probabilistic graph matching

In this paper a method is proposed to recognize symbols in electrical diagrams based on probabilistic matching. The skeletons of the symbols are represented by graphs. After finding the pose of the graph (orientation, translation, scale) by a bounded search for a minimum error transformation, the observed graph is matched to the class models and the likelihood of the match is calculated. Results are given for computer-generated symbols and hand drawn symbols with and without a template. Error rates range from <1% to 8%.     

Online region computations for Euler diagrams with relaxed drawing conventions

Euler diagrams are an accessible and effective visualisation of data involving simple set-theoretic relationships. Efficient algorithms to quickly compute the abstract regions of an Euler diagram upon curve addition and removal have previously been developed (the single marked point approach, SMPA), but a strict set of drawing conventions (called well-formedness conditions) were enforced, meaning that some abstract diagrams are not representable as concrete diagrams. We present a new methodology (the multiple marked point approach, MMPA) enabling online region computation for Euler diagrams under the relaxation of the drawing convention that zones must be connected regions. Furthermore, we indicate how to extend the methods to deal with the relaxation of any of the drawing conventions, with the use of concurrent line segments case being of particular importance. We provide complexity analysis and compare the MMPA with the SMPA. We show that these methods are theoretically no worse than other comparators, whilst our methods apply to any case, and are likely to be faster in practise due to their online nature. The machinery developed for the concurrency case could be of use in Euler diagram drawing techniques (in the context of the Euler Graph), and in computer graphics (e.g. the development of an advanced variation of a winged edge data structure that deals with concurrency). The algorithms are presented for generic curves; specialisations such as utilising fixed geometric shapes for curves may occur in applications which can enhance capabilities for fast computations of the algorithms' input structures. We provide an implementation of these algorithms, utilising ellipses, and provide time-based experimental data for benchmarking purposes.     

Online activity recognition using evolving classifiers

This paper describes the use of evolving classifiers for activity recognition from sensor readings in ambient assisted living environments. Recognizing the activities an elderly person who lives alone performs, and identifying potential problems from the detected activities is a very active topic of research. However, current approaches do not take into account the fact that the way an activity is performed by a person evolves over time and therefore activities are identified by mapping them to a static model. In this work we describe and evaluate an approach for online classifying based on Evolving Fuzzy Systems (EFS): activities are described by a model that evolves over time, according to the changes observed in the way an activity is performed. These classifiers have been evaluated on three datasets obtained from real home settings, achieving a good recognition performance, at a confidence interval of 95%, compared with well know probabilistic models in terms of F-Measure, but improving their performance in terms of online capabilities and ability to adapt to the evolving ways in which activities are carried out.     

手绘草图的在线分段识别

提出了一种在线手绘草图快速分段识别方法。先对草绘笔划进行平滑预处理,以减少噪声干扰和冗余点的数目;然后利用草绘笔划的动静态特性,以及用户的草绘习惯等特性找出笔划的关键分段点,将笔划分为多个笔划子段;最后利用基于二次曲线的方法对笔划子段进行分段识别和特征参数计算,并通过构造面向对象的图形数据结构对识别出的几何图元进行重构。实例表明,该方法具有较好的识别能力和系统稳定性,能够应用于二维概念草图的快速表达,以及后续三维草绘建模的特征笔划智能识别过程中。     

Online continual learning for human activity recognition

Sensor-based human activity recognition (HAR), with the ability to recognise human activities from wearable or embedded sensors, has been playing an important role in many applications including personal health monitoring, smart home, and manufacturing. The real-world, long-term deployment of these HAR systems drives a critical research question: how to evolve the HAR model automatically over time to accommodate changes in an environment or activity patterns. This paper presents an online continual learning (OCL) scenario for HAR, where sensor data arrives in a streaming manner which contains unlabelled samples from already learnt activities or new activities. We propose a technique, OCL-HAR, making a real-time prediction on the streaming sensor data while at the same time discovering and learning new activities. We have empirically evaluated OCL-HAR on four third-party, publicly available HAR datasets. Our results have shown that this OCL scenario is challenging to state-of-the-art continual learning techniques that have significantly underperformed. Our technique OCL-HAR has consistently outperformed them in all experiment setups, leading up to 0.17 and 0.23 improvements in micro and macro F1 scores.     

基于LEM的在线掌纹识别

主要研究利用掌纹对人进行身份鉴定。在掌纹的各种特征中，线特征是一种非常重要的特征，但由于掌纹线不规则，几乎不能用数学进行精确的刻画，因此提出了一种用直线段去近似掌纹线的方法，利用线段Hausdorff距离方法去匹配这些线段集，并对其进行了改进。实验的结果表明了该方法的有效性。     

基于B型超声的在线手势识别

开发了一种在线手势识别软件,并给出了相关在线性能评价指标.通过实验验证了B型超声人机接口的在线性能.实验结果表明:相较于现有的传统人机接口,系统对于11类手部动作具有更好的在线识别效果.平均在线动作选择时间为(0.24±0.15)s,动作完成时间为(1.27 ±0.19)s,动作完成率为(97±7)％,实时正确率为(95±5)％.     

From engineering diagrams to engineering models: Visual recognition and applications

We present a computational recognition approach to convert network-like, image-based engineering diagrams into engineering models with which computations of interests, such as CAD modeling, simulation, information retrieval and semantic-aware editing, are enabled. The proposed approach is designed to work on diagrams produced using computer-aided drawing tools or hand sketches, and does not rely on temporal information for recognition. Our approach leverages a Convolutional Neural Network (CNN) as a trainable engineering symbol recognizer. The CNN is capable of learning the visual features of the defined symbol categories from a few user-supplied prototypical diagrams and a set of synthetically generated training samples. When deployed, the trained CNN is applied either to the entire input diagram using a multi-scale sliding window or, where applicable, to each isolated pixel cluster obtained through Connected Component Analysis (CCA). Then the connectivity between the detected symbols are analyzed to obtain an attributed graph representing the engineering model conveyed by the diagram. We evaluate the performance of the approach with benchmark datasets and demonstrate its utility in different application scenarios, including the construction and simulation of control system or mechanical vibratory system models from hand-sketched or camera-captured images, content-based image retrieval for resonant circuits and sematic-aware image editing for floor plans.