智能数据分区与布局研究

大数据时代,数据规模庞大,由数据进行驱动的应用分析场景日益增多.如何快速、高效地从这些海量数据中提取出用以分析决策的信息,给数据库系统带来重大挑战.同时,现代商业分析决策对分析数据的实时性要求数据库系统能够同时快速处理ACID事务和复杂的分析查询.然而,传统的数据分区粒度太粗,且不能适应动态变化的复杂分析负载;传统的数据布局单一,不能应对现代大量增加的混合事务分析应用场景.为了解决以上问题,“智能数据分区与布局”成为当前的研究热点之一,它通过数据挖掘、机器学习等技术抽取工作负载的有效特征,设计最佳的分区策略来避免扫描大量不相关的数据,指导布局结构设计以适应不同类型的工作负载.首先介绍了智能数据分区与布局的相关背景知识,然后对智能数据分区与布局技术的研究动机、发展趋势、关键技术进行详细的阐述.最后,对智能数据分区与布局技术的研究前景做出总结与展望.

Survey of Intelligent Partition and Layout Technology in Database System

In the era of big data, there are more and more application analysis scenarios driven by large-scale data. How to quickly and efficiently extract the information for analysis and decision-making from these massive data brings great challenges to the database system. At the same time, the real-time performance of analysis data in modern business analysis and decision-making requires that the database system can process ACID transactions and complex analysis queries. However, the traditional data partition granularity is too coarse, and cannot adapt to the dynamic changes of complex analysis load; the traditional data layout is single, and cannot cope with the modern increasing mixed transaction analysis application scenarios. In order to solve the above problems, "intelligent data partition and layout" has become one of the current research hotspots. It extracts the effective characteristics of workload through data mining, machine learning, and other technologies, and design appropriate partition strategy to avoid scanning a large number of irrelevant data and guide the layout structure design to adapt to different types of workloads. This paper first introduces the background knowledge of data partition and layout techniques, and then elaborates the research motivation, development trend, and key technologies of intelligent data partition and layout. Finally, the research prospect of intelligent data partition and layout is summarized and prospected.