存储容量可扩展区块链系统的高效查询模型

区块链技术是目前计算机领域的研究热点,其实现了去中心化,并且能够安全地存储数字信息,有效降低现实经济的信任成本.提出一种区块链存储容量可扩展模型的高效查询方法——ElasticQM.此查询模型由用户层、查询层、存储层和数据层这4个模块组成.在用户层,模型将查询结果缓存,加快再次查询相同数据时的查询速度;在查询层,模型采用容量可扩展区块链模型的全局查询优化算法,增加了查询超级节点、查询验证节点和查询叶子节点这3种节点角色,提高了查询效率;在存储层,模型改进了区块链的容量可扩展模型ElasticChain的数据存储过程,实现了存储的可扩展性,并减少了占用的存储空间;在数据层,提出一种基于B-M树的区块链存储结构,并给出了B-M树的建立算法和基于B-M树的查找算法,基于B-M树的存储结构,区块链会在进行块内局部查找时提高区块链的查询速度.最后,通过在多节点不同数据量的区块链中查询的实验结果表明,ElasticQM查询方法具有高效的查询效率.     

D-Ocean: an unstructured data management system for data ocean environment

Together with the big datamovement,many organizations collect their own big data and build distinctive applications. In order to provide smart services upon big data, massive variable data should be well linked and organized to form Data Ocean, which specially emphasizes the deep exploration of the relationships among unstructured data to support smart services. Currently, almost all of these applications have to deal with unstructured data by integrating various analysis and search techniques upon massive storage and processing infrastructure at the application level, which greatly increase the difficulty and cost of application development.This paper presents D-Ocean, an unstructured data management system for data ocean environment. D-Ocean has an open and scalable architecture, which consists of a core platform, pluggable components and auxiliary tools. It exploits a unified storage framework to store data in different kinds of data stores, integrates batch and incremental processing mechanisms to process unstructured data, and provides a combined search engine to conduct compound queries. Furthermore, a so-called RAISE process modeling is proposed to support the whole process of Repository, Analysis, Index, Search and Environment modeling, which can greatly simplify application development. The experiments and use cases in production demonstrate the efficiency and usability of D-Ocean.     

基于XML的通用异构数据交换模型

为了改进传统数据交换共享平台缺乏通用性和扩展性的问题,实现企业之间业务流数据的安全交换,设计了一种基于Web服务架构的可扩展通用数据交换平台.该平台充分利用了可扩展标记语言、简单对象访问协议、统一描述、发现和集成协议及Web服务描述语言的优点,采用对称密码及非对称密码技术对企业业务数据加密,建立了Web服务器体系统结构和基于企业B2B(企业间电子商务)集成解决方案的数据交换模型,并以.NET及C#语言实现.该平台实现了企业之间异构数据独立于平台的交互,数据交换过程中具有较高的安全性.     

Scalable graph-based OLAP analytics over process execution data

In today’s knowledge-, service-, and cloud-based economy, businesses accumulate massive amounts of data from a variety of sources. In order to understand businesses one may need to perform considerable analytics over large hybrid collections of heterogeneous and partially unstructured data that is captured related to the process execution. This data, usually modeled as graphs, increasingly come to show all the typical properties of big data: wide physical distribution, diversity of formats, non-standard data models, independently-managed and heterogeneous semantics. We use the term big process graph to refer to such large hybrid collections of heterogeneous and partially unstructured process related execution data. Online analytical processing (OLAP) of big process graph is challenging as the extension of existing OLAP techniques to analysis of graphs is not straightforward. Moreover, process data analysis methods should be capable of processing and querying large amount of data effectively and efficiently, and therefore have to be able to scale well with the infrastructure’s scale. While traditional analytics solutions (relational DBs, data warehouses and OLAP), do a great job in collecting data and providing answers on known questions, key business insights remain hidden in the interactions among objects: it will be hard to discover concept hierarchies for entities based on both data objects and their interactions in process graphs. In this paper, we introduce a framework and a set of methods to support scalable graph-based OLAP analytics over process execution data. The goal is to facilitate the analytics over big process graph through summarizing the process graph and providing multiple views at different granularity. To achieve this goal, we present a model for process OLAP (P-OLAP) and define OLAP specific abstractions in process context such as process cubes, dimensions, and cells. We present a MapReduce-based graph processing engine, to support big data analytics over process graphs. We have implemented the P-OLAP framework and integrated it into our existing process data analytics platform, ProcessAtlas, which introduces a scalable architecture for querying, exploration and analysis of large process data. We report on experiments performed on both synthetic and real-world datasets that show the viability and efficiency of the approach.     

\mathcal{MD}-HBase: design and implementation of an elastic data infrastructure for cloud-scale location services

The ubiquity of location enabled devices has resulted in a wide proliferation of location based applications and services. To handle the growing scale, database management systems driving such location based services (LBS) must cope with high insert rates for location updates of millions of devices, while supporting efficient real-time analysis on latest location. Traditional DBMSs, equipped with multi-dimensional index structures, can efficiently handle spatio-temporal data. However, popular open-source relational database systems are overwhelmed by the high insertion rates, real-time querying requirements, and terabytes of data that these systems must handle. On the other hand, key-value stores can effectively support large scale operation, but do not natively provide multi-attribute accesses needed to support the rich querying functionality essential for the LBSs. We present the design and implementation of $\mathcal {MD}$ -HBase, a scalable data management infrastructure for LBSs that bridges this gap between scale and functionality. Our approach leverages a multi-dimensional index structure layered over a key-value store. The underlying key-value store allows the system to sustain high insert throughput and large data volumes, while ensuring fault-tolerance, and high availability. On the other hand, the index layer allows efficient multi-dimensional query processing. Our optimized query processing technique accesses only the index and storage level entries that intersect with the query region, thus ensuring efficient query processing. We present the design of $\mathcal {MD}$ -HBase that demonstrates how two standard index structures—the K-d tree and the Quad tree—can be layered over a range partitioned key-value store to provide scalable multi-dimensional data infrastructure. Our prototype implementation using HBase, a standard open-source key-value store, can handle hundreds of thousands of inserts per second using a modest 16 node cluster, while efficiently processing multi-dimensional range queries and nearest neighbor queries in real-time with response times as low as few hundreds of milliseconds.     

面向服务架构下的移动地理信息数据采集

传统的地理信息数据采集方式和采集平台架构制约着地理空间信息服务的发展,移动地理信息数据采集成为可行的解决手段。通过面向服务技术在移动GIS数据采集中的应用,创建易于部署和管理的客户端和服务器整体架构,采用Web Service技术创建多种地理信息服务,依托移动GIS平台实现了一系列的地图数据浏览和查询等功能。所建立的系统运行状态良好,可扩展性强,界面友好易操作,适用于各种类似的专业,为地理信息空间服务的发展提供有力的数据支撑。     

Anteater: A Service-Oriented Architecture for High-Performance Data Mining

Data mining focuses on extracting useful information from large volumes of data, and thus has been the center of much attention in recent years. Building scalable, extensible,and easy-to-use data mining systems,however,has proved to be difficult. In response, the authors developed Anteater, a service-oriented architecture for data mining that relies on Web services to achieve extensibility and interoperability, offers simple abstractions for users, and supports computationally intensive processing on large amounts of data through massive parallelism.     

流程企业数据平台的设计与实现

针对当今中国流程企业的现状，提出了一种对企业多种数据源进行整合和管理的解决方案——流程企业数据平台。该数据平台可以实现流程企业中多种数据的集成，并支持企业业务过程的持续改善和企业新应用的开发。流程企业数据平台由数据模型、数据发布接口、数据整合工具和平台管理工具组成，是平台级的软件产品。针对流程企业典型代表的石化企业的特点，给出了数据平台的实现。     

Constructing scalable Internet of Things services based on their event‐driven models

Even though many IoT (Internet of Things) applications have been implemented based on distributed events, how to construct scalable IoT services is still unclear. In this paper, we first discuss representing physical entities as IoT resources in the cyber world and integrate them into IoT services and then use an event session mechanism to express the coordination logic in an IoT service system. Given their event‐driven models, the scalable IoT services are constructed through decoupling one service at behavior from others, where running atomic service instances with high concurrency is our first focus, and distributed execution of an IoT business process is our second focus. Our last focus is to make each distributed business process execution unit hold its properties in the whole process, which requires modeling the environment as a prerequisite to compute its properties, and some assume‐guarantee rules for composing the services from the environment's perspective. We then propose a platform to support the construction, where according to the behavior decoupling features, one IoT business process is decomposed into pieces of event composition logic and business computation logic, together with separating the data dependency of computation functions in each process fragment, such that it can be concurrently executed and distributed. A practical application is implemented to concept‐prove our work. Copyright © 2015 John Wiley & Sons, Ltd.     

A flexible and scalable audio information retrieval system for mixed‐type audio signals

The content‐based classification and retrieval of real‐world audio clips is one of the challenging tasks in multimedia information retrieval. Although the problem has been well studied in the last two decades, most of the current retrieval systems cannot provide flexible querying of audio clips due to the mixed‐type form (e.g., speech over music and speech over environmental sound) of audio information in real world. We present here a complete, scalable, and extensible content‐based classification and retrieval system for mixed‐type audio clips. The system gives users an opportunity for flexible querying of audio data semantically by providing four alternative ways, namely, querying by mixed‐type audio classes, querying by domain‐based fuzzy classes, querying by temporal information and temporal relationships, and querying by example (QBE). In order to reduce the retrieval time, a hash‐based indexing technique is introduced. Two kinds of experiments were conducted on the audio tracks of the TRECVID news broadcasts to evaluate the performance of the proposed system. The results obtained from our experiments demonstrate that the Audio Spectrum Flatness feature in MPEG‐7 standard performs better in music audio samples compared to other kinds of audio samples and the system is robust under different conditions. © 2011 Wiley Periodicals, Inc.     

Experiencing OptiqueVQS: a multi-paradigm and ontology-based visual query system for end users

Data access in an enterprise setting is a determining factor for value creation processes, such as sense-making, decision-making, and intelligence analysis. Particularly, in an enterprise setting, intuitive data access tools that directly engage domain experts with data could substantially increase competitiveness and profitability. In this respect, the use of ontologies as a natural communication medium between end users and computers has emerged as a prominent approach. To this end, this article introduces a novel ontology-based visual query system, named OptiqueVQS, for end users. OptiqueVQS is built on a powerful and scalable data access platform and has a user-centric design supported by a widget-based flexible and extensible architecture allowing multiple coordinated representation and interaction paradigms to be employed. The results of a usability experiment performed with non-expert users suggest that OptiqueVQS provides a decent level of expressivity and high usability and hence is quite promising.     

Segmentifier: Interactive Refinement of Clickstream Data

Clickstream data has the potential to provide insights into e‐commerce consumer behavior, but previous techniques fall short of handling the scale and complexity of real‐world datasets because they require relatively clean and small input. We present Segmentifier, a novel visual analytics interface that supports an iterative process of refining collections of action sequences into meaningful segments. We present task and data abstractions for clickstream data analysis, leading to a high‐level model built around an iterative view‐refine‐record loop with outcomes of conclude with an answer, export segment for further analysis in downstream tools, or abandon the question for a more fruitful analysis path. Segmentifier supports fast and fluid refinement of segments through tightly coupled visual encoding and interaction with a rich set of views that show evocative derived attributes for segments, sequences, and actions in addition to underlying raw sequences. These views support fast and fluid refinement of segments through filtering and partitioning attribute ranges. Interactive visual queries on custom action sequences are aggregated according to a three‐level hierarchy. Segmentifier features a detailed glyph‐based visual history of the automatically recorded analysis process showing the provenance of each segment as an analysis path of attribute constraints. We demonstrate the effectiveness of our approach through a usage scenario with real‐world data and a case study documenting the insights gained by a corporate e‐commerce analyst.     

A self‐organized load balancing mechanism for cloud computing

The growth in computer and networking technologies over the past decades established cloud computing as a new paradigm in information technology. The cloud computing promises to deliver cost‐effective services by running workloads in a large scale data center consisting of thousands of virtualized servers. The main challenge with a cloud platform is its unpredictable performance. A possible solution to this challenge could be load balancing mechanism that aims to distribute the workload across the servers of the data center effectively. In this paper, we present a distributed and scalable load balancing mechanism for cloud computing using game theory. The mechanism is self‐organized and depends only on the local information for the load balancing. We proved that our mechanism converges and its inefficiency is bounded. Simulation results show that the generated placement of workload on servers provides an efficient, scalable, and reliable load balancing scheme for the cloud data center. Copyright © 2016 John Wiley & Sons, Ltd.     

Benchmarking performance for migrating a relational application to a parallel implementation

Many organizations rely on relational database platforms for OLAP-style querying (aggregation and filtering) for small to medium size applications. We investigate the impact of scaling up the data sizes for such queries. We intend to illustrate what kind of performance results an organization could expect should they migrate current applications to big data environments. This paper benchmarks the performance of Hive (Thusoo et al., 2009)  [9], a parallel data warehouse platform that is a part of the Hadoop software stack. We set up a 4-node Hadoop cluster using Hortonworks HDP 1.3.2 (Hortonworks HDP 1.3.2). We use the data generator provided by the TPC-DS benchmark (DSGen v1.1.0) to generate data of different scales. We compare the performance of loading data and querying for SQL and Hive Query Language (HiveQL) on a relational database installation (MySQL) and on a Hive cluster, respectively. We measure the speedup for query execution for three dataset sizes resulting from the scale up. Hive loads the large datasets faster than MySQL, while it is marginally slower than MySQL when loading the smaller datasets. Query execution in Hive is also faster. We also investigate executing Hive queries concurrently in workloads and conclude that serial execution of queries is a much better practice for clusters with limited resources.     

Heterogeneous multimedia reasoning

An innovated system is described which integrates data from heterogeneous multimedia domains. This general-purpose framework uses the concept of a `mediator' to efficiently support multimedia reasoning. The Media Abstraction Creation System (MACS) implements algorithms for creating or querying media abstractions, for relaxing queries, and for allowing incremental query modifications. We have also developed an integrated framework within which media data can be queried even when those queries need to access other (nonmedia) background data and software. Access to diverse, nonmedia databases is provided by the Heterogeneous Reasoning and Mediator System (Hermes)     

A scalable P2P platform for the knowledge grid

The knowledge grid needs to operate with a scalable platform to provide large-scale intelligent services. A key function of such a platform is to efficiently support various complex queries in a dynamic large-scale network environment. This paper proposes a platform to support index-based path queries by incorporating a semantic overlay with an underlying structured P2P network that provides object location and management services. Various distributed indexing structures can be dynamically formed by publishing, semantic objects as indexing nodes. Queries are forwarded along the chains of semantic object pointers to search for objects. We investigate the deployment of a scalable distributed trie index for broadcast queries on key strings, propose a decentralized load balancing method for solving the problem of uneven load distribution incurred by heterogeneity of loads and node capacities and by the distributed trie index, and give an approach for improving the availability of the semantic overlay and its trie index. Experiments demonstrate the scalability of the proposed platform.     

Efficient Relational Techniques for Processing Graph Queries

Graphs are widely used for modeling complicated data such as social networks, chemical compounds, protein interactions and semantic web. To effectively understand and utilize any collection of graphs, a graph database that efficiently supports elementary querying mechanisms is crucially required. For example, Subgraph and Supergraph queries are important types of graph queries which have many applications in practice. A primary challenge in computing the answers of graph queries is that pair-wise comparisons of graphs are usually hard problems. Relational database management systems (RDBMSs) have repeatedly been shown to be able to efficiently host different types of data such as complex objects and XML data. RDBMSs derive much of their performance from sophisticated optimizer components which make use of physical properties that are specific to the relational model such as sortedness, proper join ordering and powerful indexing mechanisms. In this article, we study the problem of indexing and querying graph databases using the relational infrastructure. We present a purely relational framework for processing graph queries. This framework relies on building a layer of graph features knowledge which capture metadata and summary features of the underlying graph database. We describe different querying mechanisms which make use of the layer of graph features knowledge to achieve scalable performance for processing graph queries. Finally, we conduct an extensive set of experiments on real and synthetic datasets to demonstrate the efficiency and the scalability of our techniques.     

基于SOA的海量准实时数据服务平台接入/访问研究

电网数据已成为电力企业发展的重要资产,但缺乏有效的技术手段对准实时数据进行高效的接入/访问。从松耦合、可重复性、标准化等架构设计原则考虑,通过数据接入方案和数据访问方案设计并实现了基于SOA技术的海量准实时数据服务平台。平台通过设计的数据接入/访问工具将不同业务系统的准实时数据统一接入,以测点形式有序集成,并统一对外提供订阅、查询和获取所需数据访问服务来支撑业务应用。同时通过平台电网模型同步与模型树建立和高级应用支撑的关键技术,建立主网模型,并借助变电站设备模型挂接完成主网与配网模型拼接,形成电网标准的模型树,实现电网准实时数据的高效集成和访问,更好的满足电力企业准实时数据纵向贯通和横向交互的需求。     

大数据技术在协同场景监控中的应用

跨企业信息集成（协同场景监控平台）作为大数据技术在协同场景监控中的创新应用,是企业对内部信息系统协同业务处理的监控中心。该平台整合海南电网公司内部多个信息系统中的海量协同数据,利用大数据技术进行海量数据的采集、存储、处理和展现,实现跨部门、跨业务域的数据资源流动,同时通过大数据分析的结果设计协同监控指标,快速定位各信息系统业务协同中出现的异常情况,辅助公司各级管理人员了解协同问题整体情况,跟进协同问题闭环处理过程,平台结合实际业务需求,设计开发一系列实用功能,利用平台可智能识别并自动推送的相关协同问题和解决方案,利用平台大数据可视化工具可将数据分析结果以丰富的计算机图形直观地进行展示,有效地提高协同问题解决效率,提升企业精益化管理水平。