Deriving Workloads for Performance Testing

An approach is presented to compare the performance of an existing production platform and a proposed replacement architecture. The traditional approach to such a comparison is to develop software for the proposed platform, build the new architecture, and collect performance measurements on both the existing system in production and the new system in the development environment. In this paper we propose a new way to design an application-independent workload for doing such a performance evaluation. We demonstrate the applicability of our approach by describing our experience using it to help an industrial organization determine whether or not a proposed architecture would be adequate to meet their organization's performance requirements.     

System-Level Performance Metrics for Multiprogram Workloads

Assessing the performance of multiprogram workloads running on multithreaded hardware is difficult because it involves a balance between single-program performance and overall system performance. This article argues for developing multiprogram performance metrics in a top-down fashion starting from system-level objectives. The authors propose two performance metrics: average normalized turnaround time, a user-oriented metric, and system throughput, a system-oriented metric.     

Performance Tradeoffs in Scheduling Techniques for Mixed Workloads

Many modern multimedia applications require the retrieval of different classes of data with drastically different characteristics. For instance, digital libraries type systems must be designed to deliver not only text files and still images, but voice and video as well. These applications can benefit from the sharing of resources such as disk and network bandwidth, instead of the conservative approach of partitioning the resources according to the characteristics of each type of data being retrieved. Continuous and non-continuous media applications, require different performance metrics to be achieved, so that the necessary quality of service (QoS) is satisfied. As a consequence, the proper managing of resources is a major issue in order to provide the complete sharing of resources and yet reaching the QoS goals. This work focuses on multimedia storage systems that are capable of serving a mixture of continuous and non-continuous workloads. Our main objective is to expose and investigate the tradeoffs involved in managing the system resources, in particular, I/O bandwidth. The performance metrics of interest are the mean and variance of response time for non-continuous media requests and the probability of missing an imposed deadline for continuous media workloads. Different scheduling algorithms are considered and tradeoffs to achieve performance goals are studied, including those involving buffer sizing.     

共享存储MapReduce云计算性能测试方法

为优化大量云计算线程对共享存储的访问,提出一种共享存储MapReduce云计算性能测试方法。以Oprofile为中心,对云计算的应用逻辑、动态共享库及内核系统调用进行性能统计分析,通过Valgrind与Ptrace机制完成对存储访问及系统调用的计数与计时。实验结果表明,该方法可快速定量分析共享存储的MapReduce,发现应用的内在性能瓶颈。     

用于Hadoop2.x的MapReduce性能评估模型

基于MapReduce的程序被越来越多地应用于大型数据分析的应用中.Apache Hadoop是最常用的开源MapReduce模型之一.程序运行时间的缩短对于MapReduce程序以及所有数据处理应用而言至关重要,而能够准确估算MapReduce程序的执行时间是优化程序的重要环节.本文定义了一个在Hadoop2.x版本...     

Hadoop MapReduce短作业执行性能优化

Hadoop MapReduce并行计算框架被广泛应用于大规模数据并行处理.近年来,由于其能较好地处理大规模数据,Hadoop MapReduce也被越来越多地使用在查询应用中.为了能够处理大规模数据集,Hadoop的基本设计更多地强调了数据的高吞吐率.然而在处理对短作业响应性能有较高要求的查询应用时,Hadoop MapReduce并行计算框架存在明显不足.为了提升Hadoop对于短作业的执行效率,对原有的Hadoop MapReduce作出以下3点优化：1)通过优化原有的setup和cleanup任务的执行方式,成功地缩短了作业初始化环境准备和作业结束环境清理的时间;2)将首次任务分配从“拉”模式转变为“推”模式;3)将作业执行过程中JobTracker和TaskTrackers之间的控制消息通信从现有的周期性心跳机制中分离出来,采用即时传递机制.最后,采用一种典型的基于MapReduce并行化的查询应用BLAST,对优化工作进行了评估.各种不同类型BLAST作业的测试实验表明,与现有的标准Hadoop相比,优化后的Hadoop平均执行性能提升约23%.     

Analytical Score for Multivariate GARCH Models

Multivariate GARCH models constitute the workhorse of empiricalapplications in several fields, a notable example being financialeconometrics. Unfortunately, ML (or quasi-ML) estimation of such models,although relatively straightforward in theory, is often made difficult bythe fact that available software relies on numerical methods for computingthe first derivatives of the log-likelihood; the fact that these modelsoften include a large number of parameters makes it impractical toestimate even medium-sized models. In this paper, closed-form expressionsfor the score of the BEKK model of Engle and Kroner (1995) are obtained,and strategies for efficient computation are discussed.     

Analytical Derivatives for Markov Switching Models

This paper presents analytical gradients for a broad class of regime-switching models with Markovian state-transition probabilities. Such models are usually estimated by maximum likelihood methods, which require the derivatives of the likelihood function with respect to the parameter vector. These gradients are usually calculated by means of numerical techniques. The paper shows that analytical gradients considerably speed up maximum-likelihood estimation.     

Analytical Queueing Models

Increased use of analytical performance models can be attributed, in part, to a very simple fact: they work.     

Design and Potential Performance of Goal-Oriented Job Scheduling Policies for Parallel Computer Workloads

To balance multiple scheduling performance requirements on parallel computer systems, traditional job schedulers use many parameters that can be configured to define job or queue priorities. Offering many parameters seems flexible, but in reality tuning the values for the parameters is highly challenging. To simplify the task of resource management, we propose goal-oriented policies, which allow system administrators to specify high-level performance objectives, rather than tuning low-level scheduling parameters. We study the design of goal-oriented policies, including (1) appropriate multi-objective models for specifying trade-offs between objectives, (2) efficient search algorithms for searching the best schedule at each scheduling decision point, and (3) appropriate performance measures to be optimized in the objectives with respect to two common performance requirements: preventing starvation and favoring shorter jobs. We compare goal-oriented policies with widely used backfill policies. Policies are evaluated by simulation using ten monthly workloads that ran on a Linux cluster (IA-64) from NCSA. Our results show that by automatically optimizing performance according to the given objectives through search, goal-oriented policies can simultaneously outperform FCFS-backfill and LXF-backfill, which are designed in favor of the maximum wait and average slowdown, respectively.     

Functional Models of Hadoop MapReduce with Application to Scan

MapReduce, first proposed by Google, is a remarkable programming model for processing very large amounts of data. An open-source implementation of MapReduce, called Hadoop, is now used for developing a wide range of applications. Although developing a correct and efficient program on MapReduce is much easier than developing one with MPI etc., it is still nontrivial if the target application requires involved functionalities of Hadoop MapReduce. Under these situations, functional models for MapReduce computation play important roles because we can utilize them for better understanding, proving the correctness, and even optimization of MapReduce programs. In this paper, we develop two functional models, a low-level one and a high-level one, which capture the semantics of Hadoop MapReduce computation. We discuss the detailed semantics mainly in terms of the following two computations: the computation of Mapper and Reducer classes and the computation in the Shuffle phase with the secondary-sorting technique. In addition, we develop MapReduce algorithms for the scan computational pattern (prefix sums) on the newly proposed models.     

Improving MapReduce Performance with Partial Speculative Execution

The MapReduce framework has become the de facto standard for big data processing due to its attractive features and abilities. One is that it automatically parallelizes a job into multiple tasks and transparently handles task execution on a large cluster of commodity machines. The increasing heterogeneity of distributed environments may result in a few straggling tasks, which prolong job completion. Speculative execution is proposed to mitigate stragglers. However, the existing speculative execution mechanism could not work efficiently as many speculative tasks are still slower than their original tasks. In this paper, we explore an approach to increase the efficiency of speculative execution, and further improve MapReduce performance. We propose the Partial Speculative Execution (PSE) strategy to make speculative tasks start from the checkpoint. By leveraging the checkpoint of original tasks, PSE can eliminate the costs of re-reading, re-copying, and re-computing the processed data. We implement PSE in Hadoop, and evaluate its performance in terms of job completion time and the efficiency of speculative execution under several kinds of classical workloads. Experimental results show that, in heterogeneous environments with stragglers, PSE completes jobs 56 % faster than that with no speculation and 12 % faster than that with LATE, an improved speculative execution algorithm. In addition, on average PSE can improve the efficiency of speculative execution by 24 % compared to LATE.     

IPC Considered Harmful for Multiprocessor Workloads

Many architectural simulation studies use instructions per cycle (IPC) to analyze performance. For multithreaded programs running on multiprocessor systems, however, IPC often inaccurately reflects performance and leads to incorrect or misleading conclusions. Work-related metrics, such as time per transaction, are the most accurate and reliable way to estimate multiprocessor workload performance.     

异构资源环境下的MapReduce性能优化

针对现有Hadoop难以适应异构资源环境的不足,提出一种自适应MapReduce调度器:CloudMR.基于数据局部性,CloudMR将同一机架内的对进行本地归约合并,减少中间结果中对的数目,从而减少机架间的数据传送.根据资源性能和任务特征,CloudMR动态确定节点任务槽数和数据分配量.对于计算性能高的节点,CloudMR分配较多的任务和数据量,而对于计算性能低的节点,相应地减轻任务和数据量负载.实验表明,在异构环境下,较之现有Hadoop,CloudMR减少了节点间数据传输和备份任务运行,缩短了作业完成时间.     

不同MapReduce运行系统的性能测试与分析

随着云计算技术的发展,许多MapReduce运行系统被开发出来,如Hadoop、Phoenix和Twister等.直观上,Hadoop具有很强的可扩展性、稳定性,适合处理大规模离线应用;Phoenix具有运行速度快等优点,适合处理数据密集型任务;Twister是轻量级的迭代系统,非常适合迭代式的应用.不同的应用在不同的MapReduce运行系统中有着不同的性能.通过测试不同应用在这些运行系统上的性能,给出了实验比较和性能分析,从而为大数据处理时选择合适的并行编程模型提供依据.     

Integrating Software Models and Platform Models for Performance Analysis

System performance is a key factor to take into account throughout the software life cycle of modern computer systems, mostly due to their typical characteristics such as distributed deployment, code mobility, and platform heterogeneity. An open challenge in this direction is to integrate the performance validation as a transparent and efficient activity in the system development process. Several methodologies have been proposed to automate the transformation of software/hardware models into performance models. In this paper, we do not take a transformational approach; rather, we present a framework to integrate a software model with a platform model in order to build a performance model. Performance indices are obtained from simulation of the resulting performance model. Our framework provides a library of predefined resource models, model annotation and integration procedures, and simulation support that makes the performance analysis a much easier activity. We present the results obtained from two different industrial case studies that show the maturity and the stability of our approach     

Interference Analysis of Co-Located Container Workloads: A Perspective from Hardware Performance Counters

Workload characterization is critical for resource management and scheduling.Recently,with the fast development of container technique,more and more cloud service providers like Google and Alibaba adopt containers to provide cloud services,due to the low overheads.However,the characteristics of co-located diverse services(e.g.,interactive on-line services,off-line computing services)running in containers are still not clear.In this paper,we present a comprehensive analysis of the characteristics of co-located workloads running in containers on the same server from the perspective of hardware events.Our study quantifies and reveals the system behavior from the micro-architecture level when workloads are running in different co-location patterns.Through the analysis of typical hardware events,we provide recommended/unrecommended co-location workload patterns which provide valuable deployment suggestions for datacenter administrators.     

Workloads of the Future

IT has transformed itself many times over the past decade. Yet a far more fundamental overhaul is in the making: the long-predicted world of fully ubiquitous computation and communication is finally emerging. Addressing the associated challenges and opportunities may require a fundamental rethinking of the way we do design. Most notably, the semiconductor industry must abandon its traditional component-oriented perspective and adopt a system vision. To meaningfully guide this process and fully exploit the offered opportunities, an understanding of what the workloads of the future may look like is necessary. This task is possible only through a joint effort of the application and design communities. This article characterizes the essential features of what the authors consider the dominant application classes of the future. It becomes apparent that these workloads operate under different quality metrics. Instead of performance or energy per function, metrics such as system latency, useful functionality per energy spent, and reliability and liability take center stage. The authors, therefore, issue a call to action for the creation of relevant benchmark libraries, each accompanied with a clear definition of the metrics relevant to their evaluation.