Minimizing data access latency in data grids by neighborhood‐based data replication and job scheduling

In Data Grid systems, quick data access is a challenging issue due to the high latency. The failure of requests is one of the most common matters in these systems that has an impact on performance and access delay. Job scheduling and data replication are two main techniques in reducing access latency. In this paper, we propose two new neighborhood‐based job scheduling strategies and a novel neighborhood‐based dynamic data replication algorithm (NDDR). The proposed algorithms reduce the access latency by considering a variety of practical parameters for decision making and the access delay by considering the failure probability of a node in job scheduling, replica selection, and replica placement. The proposed neighborhood concept in job scheduling includes all the nodes with low data transmission costs. Therefore, we can select the best computational node and reduce the search time by running a hierarchical and parallel search. NDDR reduces the access latency through selecting the best replica by performing a hierarchical search established based on the access time, storage queue workload, storage speed, and failure probability. NDDR improves the load balancing and data locality by selecting the best replication place considering the workload, temporal locality, geographical locality, and spatial locality. We evaluate our proposed algorithms by using Optorsim Simulator in two scenarios. The simulations confirm that the proposed algorithms improve the results compared with similar existing algorithms by 11%, 15%, 12%, and 10% in terms of mean job time, replication frequency, mean data access latency, and effective network usage, respectively.     

MapReduce optimization algorithm based on machine learning in heterogeneous cloud environment

We present an approach to optimize the MapReduce architecture, which could make heterogeneous cloud environment more stable and efficient. Fundamentally different from previous methods, our approach introduces the machine learning technique into MapReduce framework, and dynamically improve MapReduce algorithm according to the statistics result of machine learning. There are three main aspects： learning machine performance, reduce task assignment algorithm based on learning result, and speculative execution optimization mechanism. Furthermore, there are two important features in our approach. First, the MapReduce framework can obtain nodes＇ performance values in the cluster through machine learning module. And machine learning module will daily calibrate nodes＇ performance values to make an accurate assessment of cluster performance. Second, with the optimization of tasks assignment algorithm, we can maximize the performance of heterogeneous clusters. According to our evaluation result, the cluster performance could have 19% improvement in current heterogeneous cloud environment, and the stability of cluster has greatly enhanced.     

MapReduce模型的调度及容错机制研究

MapReduce是一种并行编程模型,可以用来处理和生成大量数据集。它的调度以及容错机制是模型的重要一部分。通过对MapReduce模型的执行过程进行分析,提取得到其上面的调度以及容错模型。并将P2P模型中常用的调度思想使用于MapReduce调度模型上,对原来的调度机制和容错机制做一定的修改。     

Quality of Service Aware Reliable Task Scheduling in Vehicular Cloud Computing

Vehicular Cloud Computing (VCC) facilitates real-time execution of many emerging user and intelligent transportation system (ITS) applications by exploiting under-utilized on-board computing resources available in nearby vehicles. These applications have heterogeneous time criticality, i.e., they demand different Quality-of-Service levels. In addition to that, mobility of the vehicles makes the problem of scheduling different application tasks on the vehicular computing resources a challenging one. In this article, we have formulated the task scheduling problem as a mixed integer linear program (MILP) optimization that increases the computation reliability even as reducing the job execution delay. Vehicular on-board units (OBUs), manufactured by different vendors, have different architecture and computing capabilities. We have exploited MapReduce computation model to address the problem of resource heterogeneity and to support computation parallelization. Performance of the proposed solution is evaluated in network simulator version 3 (ns-3) by running MapReduce applications in urban road environment and the results are compared with the state-of-the-art works. The results show that significant performance improvements in terms of reliability and job execution time can be achieved by the proposed task scheduling model.     

一种基于QoS的事务工作流并发调度算法

并发冲突引起的连锁夭折会降低系统性能,提出了一种基于QoS的事务工作流调度算法,该算法适应异构环境需求,支持基于QoS的延迟调度优化策略和SAFE集合扩充优化策略,可根据QoS参数调整相应的调度决策,在保证分布异构环境中复杂事务工作流并发正确性的同时减少连锁夭折.证明了算法不会引起循环等待和饿死现象,可保证调度的可串行性和可恢复性,性能模拟表明该算法适用于长期运行的事务工作流的并发调度,可有效减少连锁夭折,从而减少由此带来的性能损失.     

关于MapReduce程序运行时间的可靠估算

As data volume grows , many enterprises are considering using MapReduce for its simplicity. However, how to evaluate the performance improvement before deployment is still an issue. Current researches of MapReduce performance are mainly based on monitoring and simulation, and lack mathematical models . In this paper, we present a simple but powerful performance model for the prediction of the execution time of a MapReduce program with limited resources. We study each component of MapReduce framework, and analyze the relation between the overall performance and the number of mappers and reducers based on our model. Two typical MapReduce programs are evaluated in a small cluster with 13 nodes. Experimental results show that the mathematical performance model can estimate the execution time of MapReduce programs reliably. According to our model, number of mappers and reducers can be tuned to form a better execution pipeline and lead to better performance. The model also points out potential bottlenecks of the framework and future improvement.     

一种基于风险感知策略的多节点任务调度方法

为解决无人机(UAV)集群任务调度时面临各节点动态、不稳定的情况,该文提出一种面向多计算节点的可尽量避免任务中断且具有容错性的任务调度方法。该方法首先为基于多计算节点构建了一个以最小化任务平均完成时间为优化目标的任务分配策略;然后基于任务的完成时间和边缘计算节点的存留时间两者的概率分布,将任务计算节点上的执行风险量化成额外开销时间;最后以任务的完成时间与额外开销时间之和替换原本的完成时间,设计了风险感知的任务分配策略。在仿真环境下将该文提出的任务调度方法与3种基准调度方法进行了对比实验,实验结果表明该方法能够有效地降低任务平均响应时间、任务平均执行次数以及任务截止时间错失率。证明该文提出的方法降低了任务重调度和重新执行带来的额外开销,可实现分布式协同计算任务的调度工作,为复杂场景下的无人机集群网络提供新的技术支持。     

基于风险估计模型的网格任务动态级调度算法

网格中的各种资源的失效是不可避免的,为尽可能减少由于资源失效导致任务执行失败带来的影响,网格的任务调度算法的目标不仅要最小化任务执行的时间,还要兼顾考虑任务在资源上执行失败的风险。提出了Risk—DLS（Dynamic Level Scheduling）算法是将资源的风险估计模型与DLS算法相结合,通过仿真试验与DLS算法相比较,新算法在选择适当的参数的情况下,不仅能够最小化DAG型应用的完成时间,还能提高任务执行的成功率,有效的减少网格环境的不确定性对任务执行的影响。     

ROUTE: run‐time robust reducer workload estimation for MapReduce

MapReduce has become a popular model for large‐scale data processing in recent years. Many works on MapReduce scheduling (e.g., load balancing and deadline‐aware scheduling) have emphasized the importance of predicting workload received by individual reducers. However, because the input characteristics and user‐specified map function of a given job are unknown to the MapReduce framework before the job starts, accurately predicting workload of reducers can be a difficult challenge. To address this challenge, we present ROUTE, a run‐time robust reducer workload estimation technique for MapReduce. ROUTE progressively samples the partition size of the early completed mappers, allowing ROUTE to perform estimation at run time yet fulfilling the accuracy requirement specified by users. Moreover, by using robust estimation and bootstrapping resampling techniques, ROUTE can achieve high applicability to a wide variety of applications. Through experiments using both real and synthetic data on an 11‐node Hadoop cluster, we show ROUTE can achieve high accuracy with error rate no more than 10.92% and an improvement of 40.6% in terms of error rate while compared with the state‐of‐the‐art solution. Besides, through simulations using synthetic data, we show that ROUTE is robust to a variety of skewed distributions. Finally, we apply ROUTE to existing load balancing and deadline‐aware scheduling frameworks and show ROUTE significantly improves the performance of these frameworks. Copyright © 2016 John Wiley & Sons, Ltd.     

基于MapReduce的并行蚁群算法研究与实现

蚁群算法在处理大规模TSP问题耗时较长,为解决这一不足,给出了一种基于MapReduce编程模式的并行蚁群算法。采用MapReduce的并行优化技术对蚁群算法中最耗时的循环迭代和循环赋值部分进行改进,同时运用PC集群环境的优势将具有一定规模的小蚁群分配到对应的PC机上,使其并行执行,减少运行时间。实验证明改进后的并行蚁群算法在大数据集上运行时间明显缩短,执行效率显著提高。     

基于平衡偏斜负载方法的MapReduce性能优化机制(英文)

MapReduce has emerged as a popular computing model used in datacenters to process large amount of datasets.In the map phase,hash partitioning is employed to distribute data that sharing the same key across data center-scale cluster nodes.However,we observe that this approach can lead to uneven data distribution,which can result in skewed loads among reduce tasks,thus hamper performance of MapReduce systems.Moreover,worker nodes in MapReduce systems may differ in computing capability due to(1) multiple generations of hardware in non-virtualized data centers,or(2) co-location of virtual machines in virtualized data centers.The heterogeneity among cluster nodes exacerbates the negative effects of uneven data distribution.To improve MapReduce performance in heterogeneous clusters,we propose a novel load balancing approach in the reduce phase.This approach consists of two components:(1) performance prediction for reducers that run on heterogeneous nodes based on support vector machines models,and(2) heterogeneity-aware partitioning(HAP),which balances skewed data for reduce tasks.We implement this approach as a plug-in in current MapReduce system.Experimental results demonstrate that our proposed approach distributes work evenly among reduce tasks,and improves MapReduce performance with little overhead.     

基于预配置和配置重用的粗粒度动态可重构系统任务调度技术

配置时间过长是制约可重构系统整体性能提升的重要因素,而合理的任务调度技术可有效降低系统配置时间。该文针对粗粒度动态可重构系统(CGDRS)和具有数据依赖关系的流应用,提出了一种3维任务调度模型。首先基于该模型,设计了一种基于预配置策略的任务调度算法(CPSA);然后根据任务间的配置重用性,提出了间隔配置重用与连续配置重用策略,并据此对CPSA算法进行改进。实验结果证明,CPSA算法能够有效解决调度死锁问题、降低流应用执行时间并提高调度成功率。与其它调度算法相比,对流应用执行时间的平均优化比例达到6.13%～19.53%。     

EA-DFPSO: An intelligent energy-efficient scheduling algorithm for mobile edge networks

Cloud data centers have become overwhelmed with data-intensive applications due to the limited computational capabilities of mobile terminals. Mobile edge computing is emerging as a potential paradigm to host application execution at the edge of networks to reduce transmission delays. Compute nodes are usually distributed in edge environments, enabling crucially efficient task scheduling among those nodes to achieve reduced processing time. Moreover, it is imperative to conserve edge server energy, enhancing their lifetimes. To this end, this paper proposes a novel task scheduling algorithm named Energy-aware Double-fitness Particle Swarm Optimization (EA-DFPSO) that is based on an improved particle swarm optimization algorithm for achieving energy efficiency in an edge computing environment along with minimal task execution time. The proposed EA-DFPSO algorithm applies a dual fitness function to search for an optimal tasks-scheduling scheme for saving edge server energy while maintaining service quality for tasks. Extensive experimentation demonstrates that our proposed EA-DFPSO algorithm outperforms the existing traditional scheduling algorithms to achieve reduced task completion time and conserve energy in an edge computing environment.     

云环境下面向跨域作业的调度方法

云环境下,因数据局部性或是任务对资源的特殊偏好,一个作业所包含的任务往往需要在不同的数据中心局点上运行,此类作业称为跨域作业.跨域作业的完成时间取决于最慢任务的执行效率,即存在木桶效应.针对各域资源能力异构条件下不合理的调度策略导致跨域作业执行时间跨度过长的问题,本文提出一种面向跨域作业的启发式调度方法MIN-Max-Min,优先选择期望完成时间最短的作业执行.通过实验表明,与先来先服务的策略相比,该方法能将跨域作业平均执行时间跨度减少40%以上.     

Scientific Workflow Makespan Minimization in Edge Multiple Service Providers Environment

The edge computing model offers an ultimate platform to support scientific and real-time workflow-based applications over the edge of the network. However, scientific workflow scheduling and execution still facing challenges such as response time management and latency time. This leads to deal with the acquisition delay of servers, deployed at the edge of a network and reduces the overall completion time of workflow. Previous studies show that existing scheduling methods consider the static performance of the server and ignore the impact of resource acquisition delay when scheduling workflow tasks. Our proposed method presented a meta-heuristic algorithm to schedule the scientific workflow and minimize the overall completion time by properly managing the acquisition and transmission delays. We carry out extensive experiments and evaluations based on commercial clouds and various scientific workflow templates. The proposed method has approximately 7.7% better performance than the baseline algorithms, particularly in overall deadline constraint that gives a success rate.

     

一种引入索引结构的Apriori并行化改进算法

针对传统的Apriori算法在执行过程中存在着需要扫描多次数据库,执行效率低和需要大量的内存来存储候选项集的缺点,有人已经提出了引入索引结构的Apriori改进算法,但是该算法还是传统的Apriori算法都不适合应用到云计算平台。因此,在本文中我们提出了将引入索引结构的Apriori算法的设计思想应用到Apriori算法MapReduce并行化改进的过程中。该算法的核心思想:将数据进行MapReduce并行化分块,并在每个分块中采用索引的执行模式。该算法结合了索引结构的优点和算法并行化的优点,不仅大大提高了算法的执行速度,而且实现了在云环境下的应用。     

Highly predictable execution support for critical applications with HARETICK kernel

In this paper, the problem of providing a fully predictable execution environment for critical and hard real-time applications on embedded and DSP-based platforms is studied from the viewpoint of system architecture and operation.We introduce a set of homogenous models for time, signals and tasks, which will further serve as a basis for describing the architecture and operation of a particular hard real-time kernel – “HARETICK”. The kernel provides support for concurrent operation of hard real-time tasks (the HRT execution environment), using non-preemptive scheduling algorithms, along with soft real-time tasks (the SRT environment), using classical, preemptive, priority-based scheduling algorithms.A set of applications has been developed to test the correct operation of the HARETICK kernel according to the theoretical models and to evaluate its abilities to provide high predictability of execution for critical applications. Some of the main testing results are also discussed in the paper.     

Sustainable Spectrum Allocation Strategy for 5G Mobile Network

These days, 5G wireless communication are being created for different modern IoT (Internet of Things) applications around the world, arising with the IoT. All things considered, it is feasible to send energy efficient innovation in a manner that advances the drawn out sustainability of networks. Next-generation heterogeneous wireless communication is composed of different base stations. In this network, sustainable spectrum allocation is required to maximize the bandwidth utilization along with a reduction in power consumption. This paper proposes an algorithm for allocating an optimized spectrum to clusters in a multi-cluster environment for sustainable 5G environment using particle swarm optimization (PSO). The proposed strategy is applicable for 3G, 4G, and 5G mobile networks. Mobile devices enter and leave the cluster randomly and stay within the cluster for an uncertain amount of time. During that period the user demands may vary. Consequently, various bandwidth allocations are required. For such cases, static allocation might result in inefficient utilization of bandwidth, wastage of power, and degrade user satisfaction. The proposed algorithm will optimize the spectrum allocated to a cluster from time to time to solve this problem and produce an optimized solution within a given deadline. PSO based proposed scalable spectrum allocation method is applicable for the different frequency range for each cluster, hence scalable from 3G telecommunication to 5G-mobile edge technology. The convergence of the strategy is analyzed. From simulation analysis, it is observed that the proposed strategy reduces power consumption by approximately 8%, 11%, and 6% in 3G, 4G, and 5G communications respectively than the existing scheme.

     

基于MapReduce虚拟机的Deep Web数据源发现方法

为了提高Deep Web爬虫发现和搜集数据源的效率,提出了一种融合MapReduce和虚拟化技术实现DeepWeb海量数据挖掘的并行计算方法。基于MapReduce架构提出了一个Deep Web爬虫模型,通过链接过滤分类、页面过滤分类、表单过滤分类等3个MapReduce过程找到Deep Web数据源接口,并利用虚拟机构建单机集群进行性能测试。实验结果显示该方法可以实现大规模数据的并行处理,有效提高爬虫数据源发现的效率,避免网络及物理资源的浪费,验证了云计算技术在Deep Web数据挖掘方面的可行性。     

Hierarchical power management of a system with autonomously power-managed components using reinforcement learning

This paper presents a hierarchical dynamic power management (DPM) framework based on reinforcement learning (RL) technique, which aims at power savings in a computer system with multiple I/O devices running a number of heterogeneous applications. The proposed framework interacts with the CPU scheduler to perform effective application-level scheduling, thereby enabling further power savings. Moreover, it considers non-stationary workloads and differentiates between the service request generation rates of various software application. The online adaptive DPM technique consists of two layers: component-level local power manager and system-level global power manager. The component-level PM policy is pre-specified and fixed whereas the system-level PM employs temporal difference learning on semi-Markov decision process as the model-free RL technique, and it is specifically optimized for a heterogeneous application pool. Experiments show that the proposed approach considerably enhances power savings while maintaining good performance levels. In comparison with other reference systems, the proposed RL-based DPM approach, further enhances power savings, performs well under various workloads, can simultaneously consider power and performance, and achieves wide and deep power-performance tradeoff curves. Experiments conducted with multiple service providers confirm that up to 63% maximum energy saving per service provider can be achieved.