高强度I/O的应用对并行存储系统的挑战和解决方法研究

具有高I/O密集特性的高性能计算应用对高性能计算机存储系统综合性能的要求越来越高.以石油地震勘探数据处理为代表的一类重要应用表现出I/O数据量巨大、I/O访问密度大,对单个磁盘阵列存储部件的读写带宽要求高的特征.在Lustre文件系统中,充当对象存储服务功能的磁盘阵列设备输出带宽的不足将成为阻碍存储系统整体性能发挥的重要因素.针对此类问题,提出了一种缓存管理方法,分别在客户端添加VDISK模块,在OST端添加Cache模块,二者协同提高并行文件系统I/O的输出带宽的使用效率;另外,充分利用客户端空闲内存以及客户端之间的通信带宽,降低应用程序对磁盘阵列设备输出带宽的要求.通过大规模并行模型的验证表明,VDISK提高了实际可用的输出带宽,提高了外部存储系统的I/O效率.

Challenges and Solutions of Parallel Storage Systems for I/O-Intensive Applications

I/O-intensive high-performance computing applications raise higher and higher demand for supercomputer I/O system performance. Petroleum seismic exploration data processing, as the representative of an important class of applications, declares features of huge I/O amount of data, large access density, and high bandwidth requirements for single I/O storage unit. The lack of Lustre file system OST’s physical disk arrays bandwidth become an obstacle to the overall system performance. For the features of uneven utilization of the system resource, we propose a mechanism that adding VDISK module in the Client and Cache module on the OST side, whose coordination would improve output bandwidth of the parallel file system. And the use of free memory in the Client and the communication bandwidth among Clients could reduce the requirements for the disk array output bandwidth. After experiments that use large-scale parallel model, it is validated that VDISK increases the output of the actual available bandwidth, reaches to the improvement of the overall system I/O efficiency.