MARVIN: a medical research application framework based on open source software

This paper describes the open source framework MARVIN for rapid application development in the field of biomedical and clinical research. MARVIN applications consist of modules that can be plugged together in order to provide the functionality required for a specific experimental scenario. Application modules work on a common patient database that is used to store and organize medical data as well as derived data. MARVIN provides a flexible input/output system with support for many file formats including DICOM, various 2D image formats and surface mesh data. Furthermore, it implements an advanced visualization system and interfaces to a wide range of 3D tracking hardware. Since it uses only highly portable libraries, MARVIN applications run on Unix/Linux, Mac OS X and Microsoft Windows.     

手持移动设备中多通道交互的通用开发框架

提出的多通道交互通用开发框架可以快速地开发出支持多通道交互的应用,也可以将已经存在的应用快速更改为多通道交互应用;基于此理论,设计和实现了一种支持手写笔和语音输入的多通道家电开关控制系统.具体地阐明了实际开发中应该重点注意的环节,希望能够对当前手持移动设备的多通道交互化有所借鉴.     

EmbraceNet: A robust deep learning architecture for multimodal classification

Classification using multimodal data arises in many machine learning applications. It is crucial not only to model cross-modal relationship effectively but also to ensure robustness against loss of part of data or modalities. In this paper, we propose a novel deep learning-based multimodal fusion architecture for classification tasks, which guarantees compatibility with any kind of learning models, deals with cross-modal information carefully, and prevents performance degradation due to partial absence of data. We employ two datasets for multimodal classification tasks, build models based on our architecture and other state-of-the-art models, and analyze their performance on various situations. The results show that our architecture outperforms the other multimodal fusion architectures when some parts of data are not available.     

PolSAR image classification via multimodal sparse representation-based feature fusion

ABSTRACT

It is well known that various features extraction approaches are utilized in polarimetric synthetic aperture (PolSAR) terrain classification for representing the data characteristic. It needs relevant and effective feature fusion algorithms to process complicated features. To address this issue, this article presents a multimodal sparse representation (MSR) framework based algorithm to fuse the different feature vectors from the complicated data space. Polarimetric data features, decomposition features, and the texture features from Pauli colour-coded image are selected to represent multimodal data in different observation modes. The corresponding multimodal manifold regularizations are added to MSR framework to approximate the data structure. Considering the independence and correlation of features, the intrinsic affinity matrices are calculated from this framework. They are processed via local preserve projection algorithm to project the multimodal features into a low dimensionally intrinsic feature space for subsequent classification. Three datasets are utilized in experiments, Western Xi’an, Flevoland, and San Francisco Bay datasets from the Radarsat-2 system in C-band. The effect of regularization parameters and different dimensional fused features are analysed in visualization and quantitation performance. The experiment results demonstrate that the effectiveness and validity of proposed method are superior to other state-of-art methods.     

元宇宙多模态触觉感知与交互的数据手套技术研究

为了提升元宇宙虚拟手交互的逼真性,研发了一种多模态触觉感知与交互的数据手套系统,该系统具有硬件集成度高、响应速度快、数据采集精确和轻便易穿戴的特点。人手穿戴此数据手套后,系统能够采集人手操作时的温度、指尖按压力和手指弯曲程度等数据,并把这些数据实时赋予给虚拟环境中的虚拟手,让虚拟手具有类似人手的动作行为和感知。提出了一种多通道循环采集数据方法,该方法可以高效且准确地捕捉温度、指尖按压力和手指弯曲程度等异构多模态数据,减轻了数据手套的硬件电路复杂性。实验结果表明,此数据手套在获取实时手指动作信息方面精度较高,提升了虚拟手动作的灵活性、直观性和自然性。     

数据受限条件下的多模态处理技术综述

随着多媒体技术的发展,可获取的媒体数据在种类和量级上大幅提升。受人类感知方式的启发,多种媒体数据互相融合处理,促进了人工智能在计算机视觉领域的研究发展,在遥感图像解译、生物医学和深度估计等方面有广泛的应用。尽管多模态数据在描述事物特征时具有明显优势,但仍面临着较大的挑战。1）受到不同成像设备和传感器的限制,难以收集到大规模、高质量的多模态数据集;2）多模态数据需要匹配成对用于研究,任一模态的缺失都会造成可用数据的减少;3）图像、视频数据在处理和标注上需要耗费较多的时间和人力成本,这些问题使得目前本领域的技术尚待攻关。本文立足于数据受限条件下的多模态学习方法,根据样本数量、标注信息和样本质量等不同的维度,将计算机视觉领域中的多模态数据受限方法分为小样本学习、缺乏强监督标注信息、主动学习、数据去噪和数据增强5个方向,详细阐述了各类方法的样本特点和模型方法的最新进展。并介绍了数据受限前提下的多模态学习方法使用的数据集及其应用方向（包括人体姿态估计、行人重识别等）,对比分析了现有算法的优缺点以及未来的发展方向,对该领域的发展具有积极的意义。     

A knowledge-based framework for intelligent-data migration

Abstract: The Object-Inferencing Framework (OIF) is a knowledge-based system developed for intelligent-data migration. The framework provides a mechanism to integrate relational data which represents a source model; a project-specific rulebase which specifies plausible migration scenarios; and a deduction system to facilitate the migration of source data to a new, complex target model. Typically, the target model includes constituents that possess both graphic and tabular components. Although the framework is experimental, industrial applications built upon OIF have been successfully deployed in scenarios in which the source data contained implicit information in that semantic relationships and structure conveyed by the data had to be inferred by a domain expert. This framework provides a substrate for migration from any unstructured or semi-structured data representation to a complex, semantically rich target data model. Examples of the migration of CAD data, which represents an electrical-distribution system, to a client-server based Automated Mapping/Facilities Management (AM/FM) platform are presented to convey the salient features of the design and utility of the OIF. Even though the examples are taken from a specific domain, the approach has potential applications in a myriad of domains, including business enterprises in which the migration of data created and managed by legacy systems to object-oriented and clientserver environments is an area of intense research and development.     

VoiceXML dialog system of the multimodal IP-Telephony—The application for voice ordering service

The development of IP-Telephony in recent years has been substantial. The improvement in voice quality, the integration between voice and data, especially the interaction with multimedia has made the 3G communication more promising. The value added services of Telephony techniques alleviate the dependence on the phone and provide a universal platform for the multimodal telephony applications. For example, the web-based application with VoiceXML has been developed to simplify the human–machine interaction because it takes the advantage of the speech-enabled services and makes the telephone-web access a reality. However, it is not cost-efficient to build voice only stand-alone web application and is more reasonable that voice interfaces should be retrofitted to be compatible or collaborate with the existing HTML or XML-based web applications. Therefore, this paper considers that the functionality of the web service should enable multiple access modalities so that users can perceive and interact with the site in either visual or speech response simultaneously. Under this principle, our research develops a prototype system of multimodal VoIP with the integrated web-based Mandarin dialog system which adopts automatic speech recognition (ASR), text-to-speech (TTS), VoiceXML browser, and VoIP technologies to create user friendly graphic user interface (GUI) and voice user interface (VUI). The users can use traditional telephone, cellular phone, or even VoIP connection via personal computer to interact with the VoiceXML server. In the mean time, the users browse the web and access the same content with common HTML or XML-based browser. The proposed system shows excellent performance and can be easily incorporated into voice ordering service for a wider accessibility.     

Quantum-inspired multimodal fusion for video sentiment analysis

We tackle the crucial challenge of fusing different modalities of features for multimodal sentiment analysis. Mainly based on neural networks, existing approaches largely model multimodal interactions in an implicit and hard-to-understand manner. We address this limitation with inspirations from quantum theory, which contains principled methods for modeling complicated interactions and correlations. In our quantum-inspired framework, the word interaction within a single modality and the interaction across modalities are formulated with superposition and entanglement respectively at different stages. The complex-valued neural network implementation of the framework achieves comparable results to state-of-the-art systems on two benchmarking video sentiment analysis datasets. In the meantime, we produce the unimodal and bimodal sentiment directly from the model to interpret the entangled decision.     

Saliency-weighted graphs for efficient visual content description and their applications in real-time image retrieval systems

The exponential growth in the volume of digital image databases is making it increasingly difficult to retrieve relevant information from them. Efficient retrieval systems require distinctive features extracted from visually rich contents, represented semantically in a human perception-oriented manner. This paper presents an efficient framework to model image contents as an undirected attributed relational graph, exploiting color, texture, layout, and saliency information. The proposed method encodes salient features into this rich representative model without requiring any segmentation or clustering procedures, reducing the computational complexity. In addition, an efficient graph-matching procedure implemented on specialized hardware makes it more suitable for real-time retrieval applications. The proposed framework has been tested on three publicly available datasets, and the results prove its superiority in terms of both effectiveness and efficiency in comparison with other state-of-the-art schemes.     

General-purpose modular hardware and software framework for mobile outdoor augmented reality applications in engineering

This paper presents a reusable, general-purpose, mobile augmented reality (AR) framework developed to address the critical and repetitive challenges specific to visualization in outdoor AR. In all engineering applications of AR developed thus far, basic functionality that supports accurate user registration, maximizes the range of user motion, and enables data input and output has had to be repeatedly re-implemented. This is primarily due to the fact that designed methods have been traditionally custom created for their respective applications and are not generic enough to be readily shared and reused by others. The objective of this research was to remedy this situation by designing and implementing a reusable and pluggable hardware and software framework that can be used in any AR application without the need to re-implement low-level communication interfaces with selected hardware. The underlying methods of hardware communication as well as the object-oriented design (OOD) of the reusable interface are presented. Details on the validation of framework reusability and pluggability are also described.     

Safety-critical systems built with COTS

In the rail transportation industry competitive pressure has led to the increased use of COTS (commercial off-the-shelf equipment in safety critical systems), making it imperative that we extend proven safety techniques to COTS based systems as well. To this end, we have developed the Vital Framework (V-Frame), which is used to develop a safety critical platform from COTS hardware and software. The key technologies in this framework are formal methods, information redundancy, a proprietary data format, and a concurrent checking scheme. Combining these technologies results in a real time, checkable correctness criterion that is a signature of the application's algorithm structure and is independent of both the hardware and the operating system. V-Frame's most significant attribute is that the fail safe properties of applications do not require the firmware to be correct: the application will operate in a fail safe (or vital) manner even if there are design faults in the operating system and/or the hardware fails. This does not mean that the application does not have to be correctly specified and designed. Formal methods are appropriate in the design of safety critical COTS systems because a generic processing environment is analogous to a formal system: it is designed to apply well defined transformation rules to inputs     

基于构件的医疗保险领域应用框架技术研究

本文简要介绍了基于构件的软件开发方法(CBD)和基于构件的业务应用软件框架(CBAF),以此为基础建立了一个能满足分布异构环境系统集成需要、具备软件互操作能力、实现Client/Server与Browser/Server应用相统一的医疗保险领域应用框架,并对涉及的关键技术进行了论述。     

一种业务逻辑可重构的三层应用服务器设计与实现

将网络应用系统的逻辑抽象为表单流、事件流和数据流,提出了一种业务逻辑可重构的三层应用服务器框架,由此开发了具有可重构能力的企业信息化综合业务处理系统(ICETIP)应用服务器。ICETIP应用服务器实现了客户连接池和数据库连接池的协调工作,给出了一种客户端和服务器之间通信机制。ICETIP系统的主要特点是提出了一种全新的网络应用开发与维护模式,在不需要编程的条件下可方便地依照应用逻辑构造不同行业需求的网络应用系统。     

A framework for bimanual inter-device interactions

A shared interactive display (e.g., a tabletop) provides a large space for collaborative interactions. However, a public display lacks a private space for accessing sensitive information. On the other hand, a mobile device offers a private display and a variety of modalities for personal applications, but it is limited by a small screen. We have developed a framework that supports fluid and seamless interactions among a tabletop and multiple mobile devices. This framework can continuously track each user’s action (e.g., hand movements or gestures) on top of a tabletop and then automatically generate a unique personal interface on an associated mobile device. This type of inter-device interactions integrates a collaborative workspace (i.e., a tabletop) and a private area (i.e., a mobile device) with multimodal feedback. To support this interaction style, an event-driven architecture is applied to implement the framework on the Microsoft PixelSense tabletop. This framework hides the details of user tracking and inter-device communications. Thus, interface designers can focus on the development of domain-specific interactions by mapping user’s actions on a tabletop to a personal interface on his/her mobile device. The results from two different studies justify the usability of the proposed interaction.     

Scalable and Adaptive Joins for Trajectory Data in Distributed Stream System

As a fundamental operation in LBS (location-based services), the trajectory similarity of moving objects has been extensively studied in recent years. However, due to the increasing volume of moving object trajectories and the demand of interactive query performance, the trajectory similarity queries are now required to be processed on massive datasets in a real-time manner. Existing work has proposed distributed or parallel solutions to enable large-scale trajectory similarity processing. However, those techniques cannot be directly adapted to the real-time scenario as it is likely to generate poor balancing performance when workload variance occurs on the incoming trajectory stream. In this paper, we propose a new workload partitioning framework, ART (Adaptive Framework for Real-Time Trajectory Similarity), which introduces practical algorithms to support dynamic workload assignment for RTTS (real-time trajectory similarity). Our proposal includes a processing model tailored for the RTTS scenario, a load balancing framework to maximize throughput, and an adaptive data partition manner designed to cut off unnecessary network cost. Based on this, our model can handle the large-scale trajectory similarity in an on-line scenario, which achieves scalability, effectiveness, and efficiency by a single shot. Empirical studies on synthetic data and real-world stream applications validate the usefulness of our proposal and prove the huge advantage of our approach over state-of-the-art solutions in the literature.

     

基于多组件融合与空洞图卷积的车道占用率预测模型

为解决交通拥堵和交通硬件资源分配不足等问题,提出一种基于多组件融合与空洞图卷积的车道占用率预测模型MCFDGCN。针对交通数据的非线性和受多种隐式因素影响的特点,利用图卷积提取交通数据的空间相关性,使用空洞卷积提取时间依赖特征,将车流量和车辆速度作为2个隐式因素引入模型中,对多组件提取的影响车道占用率的多模态特征进行融合,以完成车道占用率预测任务。在PeMS7（O）、PeMS7（4）数据集上进行实验,结果表明,与HA、ARIMA等模型相比,MCFDGCN模型预测误差较低且误差增长较缓慢,能实现更精准的车道占用率预测。     

Techniques for Computing Fitness of Use (FoU) for Time Series Datasets with Applications in the Geospatial Domain

Time series data are widely used in many applications including critical decision support systems. The goodness of the dataset, called the Fitness of Use (FoU), used in the analysis has direct bearing on the quality of the information and knowledge generated and hence on the quality of the decisions based on them. Unlike traditional quality of data which is independent of the application in which it is used, FoU is a function of the application. As the use of geospatial time series datasets increase in many critical applications, it is important to develop formal methodologies to compute their FoU and propagate it to the derived information, knowledge and decisions. In this paper we propose a formal framework to compute the FoU of time series datasets. We present three different techniques using the Dempster–Shafer belief theory framework as the foundation. These three approaches investigate the FoU by focusing on three aspects of data: data attributes, data stability, and impact of gap periods, respectively. The effectiveness of each approach is shown using an application in hydrological datasets that measure streamflow. While we use hydrological information analysis as our application domain in this research, the techniques can be used in many other domains as well.

Ensemble-based regression analysis of multimodal medical data for osteopenia diagnosis

Areal bone mineral density (aBMD) is used in clinical practice to diagnose osteoporosis. In previous studies, aBMD was estimated from diagnostic computed tomography (dCT) images, but a battery of medical tests was also taken that can be used to improve the regression performance. However, it is difficult to exploit the multimodal data as the additional features have poor informativeness and may lead to overfitting. An ensemble-based framework is proposed to improve the regression accuracy and robustness on multimodal medical data with a high relative dimensionality. Instead of case-wise bootstrap aggregating, a filtering-based metalearner scheme was employed to build feature-wise ensembles. The proposed approach was evaluated on clinical data and was found to be superior to bagging and other ensemble methods. The feature-wise ensembling approach can also be used to automatically determine if any multimodal features are related to bone mineral density. Several blood measurements were identified to be linked with bone mineral density, and a literature search supported the automatic identification results.     

融合多模态数据的人体动作识别方法研究

基于多模态融合的人体动作识别技术被广泛研究与应用,其中基于特征级或决策级的融合是在单一级别阶段下进行的,无法将真正的语义信息从数据映射到分类器。提出一种多级多模态融合的人体动作识别方法,使其更适应实际的应用场景。在输入端将深度数据转换为深度运动投影图,并将惯性数据转换成信号图像,通过局部三值模式分别对深度运动图和信号图像进行处理,使每个输入模态进一步转化为多模态。将所有的模态通过卷积神经网络训练进行提取特征,并把提取到的特征通过判别相关分析进行特征级融合。利用判别相关分析最大限度地提高两个特征集中对应特征的相关性,同时消除每个特征集中不同类之间的特征相关性,将融合后的特征作为多类支持向量机的输入进行人体动作识别。在UTD-MHAD和UTD Kinect V2 MHAD两个多模态数据集上的实验结果表明,多级多模态融合框架在两个数据集上的识别精度分别达到99.8%和99.9%,具有较高的识别准确率。