首页 | 官方网站   微博 | 高级检索  
     


Enhancing throughput of the Hadoop Distributed File System for interaction-intensive tasks
Authors:Xiayu Hua  Hao Wu  Zheng Li  Shangping Ren
Affiliation:Department of Computer Science, Illinois Institute of Technology, 10 West 31st, Chicago, IL, USA
Abstract:The Hadoop Distributed File System (HDFS) is designed to run on commodity hardware and can be used as a stand-alone general purpose distributed file system (Hdfs user guide, 2008). It provides the ability to access bulk data with high I/O throughput. As a result, this system is suitable for applications that have large I/O data sets. However, the performance of HDFS decreases dramatically when handling the operations of interaction-intensive files, i.e., files that have relatively small size but are frequently accessed. The paper analyzes the cause of throughput degradation issue when accessing interaction-intensive files and presents an enhanced HDFS architecture along with an associated storage allocation algorithm that overcomes the performance degradation problem. Experiments have shown that with the proposed architecture together with the associated storage allocation algorithm, the HDFS throughput for interaction-intensive files increases 300% on average with only a negligible performance decrease for large data set tasks.
Keywords:HDFS  Interaction intensive task  Cache  Hierarchical structure  PSO  Storage allocation algorithm
本文献已被 ScienceDirect 等数据库收录!
设为首页 | 免责声明 | 关于勤云 | 加入收藏

Copyright©北京勤云科技发展有限公司    京ICP备09084417号-23

京公网安备 11010802026262号