基于MapReduce的并行蚁群算法研究与实现

蚁群算法在处理大规模TSP问题耗时较长,为解决这一不足,给出了一种基于MapReduce编程模式的并行蚁群算法。采用MapReduce的并行优化技术对蚁群算法中最耗时的循环迭代和循环赋值部分进行改进,同时运用PC集群环境的优势将具有一定规模的小蚁群分配到对应的PC机上,使其并行执行,减少运行时间。实验证明改进后的并行蚁群算法在大数据集上运行时间明显缩短,执行效率显著提高。     

Performance Improvement of MapReduce for Heterogeneous Clusters Based on Efficient Locality and Replica Aware Scheduling (ELRAS) Strategy

MapReduce is a parallel programming model for processing the data-intensive applications in a cloud environment. The scheduler greatly influences the performance of MapReduce model while utilized in heterogeneous cluster environment. The dynamic nature of cluster environment and computing workloads affect the execution time and computational resource usage in the scheduling process. Further, data locality is essential for reducing total job execution time, cross-rack communication, and to improve the throughput. In the present work, a scheduling strategy named efficient locality and replica aware scheduling (ELRAS) integrated with an autonomous replication scheme (ARS) is proposed to enhance the data locality and performs consistently in the heterogeneous environment. ARS autonomously decides the data object to be replicated by considering its popularity and removes the replica as it is idle. The proposed approach is validated in a heterogeneous cluster environment with various realistic applications that are IO bound, CPU bound and mixed workloads. ELRAS improves the throughput by a factor about 2 as compared with the existing FIFO and it also yields near optimal data locality, reduce the execution time, and effective utilization of resources. The simplicity of ELRAS algorithm proves its feasibility to adopt for a wide range of applications.

     

基于平衡偏斜负载方法的MapReduce性能优化机制(英文)

MapReduce has emerged as a popular computing model used in datacenters to process large amount of datasets.In the map phase,hash partitioning is employed to distribute data that sharing the same key across data center-scale cluster nodes.However,we observe that this approach can lead to uneven data distribution,which can result in skewed loads among reduce tasks,thus hamper performance of MapReduce systems.Moreover,worker nodes in MapReduce systems may differ in computing capability due to(1) multiple generations of hardware in non-virtualized data centers,or(2) co-location of virtual machines in virtualized data centers.The heterogeneity among cluster nodes exacerbates the negative effects of uneven data distribution.To improve MapReduce performance in heterogeneous clusters,we propose a novel load balancing approach in the reduce phase.This approach consists of two components:(1) performance prediction for reducers that run on heterogeneous nodes based on support vector machines models,and(2) heterogeneity-aware partitioning(HAP),which balances skewed data for reduce tasks.We implement this approach as a plug-in in current MapReduce system.Experimental results demonstrate that our proposed approach distributes work evenly among reduce tasks,and improves MapReduce performance with little overhead.     

基于Hadoop的分布式哈希算法实现

互联网的发展使得计算密集型的任务正在逐渐走向分布式和云计算。文中对Hadoop项目中的MapReduce和HDFS进行了研究,采用HDFS作为底层分布式文件系统,MapReduce作为编程框架来实现哈希算法。通过对多个节点中的测试结果的分析表明,在Hadoop上运行哈希函数的任务,能够起到在多台计算机的群集中分摊负载的效果,并且有效地减少了任务的总时间开销。以Hadoop为基础的云计算平台具有良好的可靠性和可扩展性,对于哈希算法在Hadoop平台上的实现和测试,为将来密钥恢复等系统的研究和搭建提供了良好的基础。     

基于Hadoop的云平台设计与实现

Hadoop是一个免费、可靠、高效、可扩展的开源云平台,允许在分布式集群上处理大数据的软件框架。本文以Hadoop为基础,详细介绍了虚拟机VMware、JDK、CentOS、Hadoop等技术。在伪分布式环境下搭建虚拟云平台,经过测试,本系统能正常运行MapReduce化的分布式程序,本文还针对用户权限、路径配置和使用SSH服务程序等问题进行了详细的阐述,为基于Hadoop的云平台研究和应用程序开发提供了基础。     

Online CQI‐based optimization using k‐means and machine learning approach under sparse system knowledge

Modern cellular mobile networks are becoming more complicated and too expensive in terms of deployment, operation, and maintenance. Traffic demand in cellular networks typically experiences spatio‐temporal variations because of users' mobility and usage behaviour, which lead some of the cells to get overloaded without fully utilizing network capacity. To tackle these challenges, nowadays, self‐organizing networks (SONs) become an essential feature. This paper offers the development of an optimization framework for SONs based on channel quality indicator (CQI) and loading condition without detail knowledge of the network environment. Since the electrical tilt plays a key role in optimizing both coverage and capacity, the main motive is to ensure efficient network operation by electrical tilt‐based radio frequency (RF) performance optimization using a machine learning approach. This novel methodology shows two‐step optimization algorithms: (a) cluster formation based on handover success rate using k‐means algorithm and (b) reinforcement learning‐based optimization. Simulation and field trial shows that the proposed approach provides better results than the conventional method of prediction using genetic algorithm (GA) and other online approaches.     

Internet of things intrusion detection model and algorithm based on cloud computing and multi-feature extraction extreme learning machine

With the rapid development of the Internet of Things (IoT), there are several challenges pertaining to security in IoT applications. Compared with the characteristics of the traditional Internet, the IoT has many problems, such as large assets, complex and diverse structures, and lack of computing resources. Traditional network intrusion detection systems cannot meet the security needs of IoT applications. In view of this situation, this study applies cloud computing and machine learning to the intrusion detection system of IoT to improve detection performance. Usually, traditional intrusion detection algorithms require considerable time for training, and these intrusion detection algorithms are not suitable for cloud computing due to the limited computing power and storage capacity of cloud nodes; therefore, it is necessary to study intrusion detection algorithms with low weights, short training time, and high detection accuracy for deployment and application on cloud nodes. An appropriate classification algorithm is a primary factor for deploying cloud computing intrusion prevention systems and a prerequisite for the system to respond to intrusion and reduce intrusion threats. This paper discusses the problems related to IoT intrusion prevention in cloud computing environments. Based on the analysis of cloud computing security threats, this study extensively explores IoT intrusion detection, cloud node monitoring, and intrusion response in cloud computing environments by using cloud computing, an improved extreme learning machine, and other methods. We use the Multi-Feature Extraction Extreme Learning Machine (MFE-ELM) algorithm for cloud computing, which adds a multi-feature extraction process to cloud servers, and use the deployed MFE-ELM algorithm on cloud nodes to detect and discover network intrusions to cloud nodes. In our simulation experiments, a classical dataset for intrusion detection is selected as a test, and test steps such as data preprocessing, feature engineering, model training, and result analysis are performed. The experimental results show that the proposed algorithm can effectively detect and identify most network data packets with good model performance and achieve efficient intrusion detection for heterogeneous data of the IoT from cloud nodes. Furthermore, it can enable the cloud server to discover nodes with serious security threats in the cloud cluster in real time, so that further security protection measures can be taken to obtain the optimal intrusion response strategy for the cloud cluster.     

基于MapReduce的PageRank算法的研究

在对PageRank算法进行研究的基础上,利用MapReduce编程模型思想对PageRank算法进行改进,设计了在云平台Hadoop环境下运行的基于MapReduce的PageRank分布式并行算法,并在实验中对不同规模的Web图数据集进行了测试,分析不同的Blocksize参数对于算法计算性能的作用以及集群节点数目对于算法运行效率的影响.     

Scheduling Hardware-Accelerated Cloud Functions

This paper presents a Function-as-a-Service (FaaS) approach for deploying managed cloud functions onto heterogeneous cloud infrastructures. Current FaaS systems, such as AWS Lambda, allow domain-specific functionality, such as AI, HPC and image processing, to be deployed in the cloud while abstracting users from infrastructure and platform concerns. Existing approaches, however, use a single type of resource configuration to execute all function requests. In this paper, we present a novel FaaS approach that allows cloud functions to be effectively executed across heterogeneous compute resources, including hardware accelerators such as GPUs and FPGAs. We implement heterogeneous scheduling to tailor resource selection to each request, taking into account performance and cost concerns. In this way, our approach makes use of different processor types and quantities (e.g. 2 CPU cores), uniquely suited to handle different types of workload, potentially providing improved performance at a reduced cost. We validate our approach in three application domains: machine learning, bio-informatics, and physics, and target a hardware platform with a combined computational capacity of 24 FPGAs and 12 CPU cores. Compared to traditional FaaS, our approach achieves a cost improvement for non-uniform traffic of up to 8.9 times, while maintaining performance objectives.

     

基于MapReduce技术的并行集成分类算法

由于计算机内存资源限制,分类器组合的有效性及最优性选择是机器学习领域的主要研究内容。经典的集成分类算法在处理小数据集时,拥有较高的分类准确性,但面对大量数据时,由于多基分类器学习、分类共用1台计算机资源,导致运算效率较低,这显然不适合处理当今的海量数据。针对已有集成分类算法只适合作用于小规模数据集的缺点,剖析了集成分类器的特性,采用基于聚合方式的集成分类器和云计算的MapReduce技术设计了并行集成分类算法(EMapReduce),达到并行处理大规模数据的目的。并在Amazon计算集群上模拟实验,实验结果表明该算法具有一定的高效性和可行性。     

ACPSO—SVR结合的非线性建模预测算法研究

提出一种基于自适应混沌粒子群优化和支持向量机结合的非线性预测建模算法（ACPSO—SVR）,引入ACPSO启发式寻优机制对SVR模型的超参数进行自动选取,在超参数取值范围变化较大的情况下,效果明显优于网格式搜索算法。选取UCI机器学习数据库中的Forestfires标准数据集进行测试,实验结果表明该方法具有较高的精度和良好的泛化能力．对于解决多变量的回归预测问题是一种有效的方法。最后给出了混合算法在碳一多相催化领域的两种典型应用．在反应动力学模型未知的情况下建立催化剂组份模型和操作条件模型,以及基于混合算法的最优催化剂设计框架。     

A Lion‐Whale optimization‐based migration of virtual machines for data centers in cloud computing

The scalability, reliability, and flexibility in the cloud computing services are the obligations in the growing demand of computation power. To sustain the scalability, a proper virtual machine migration (VMM) approach is needed with apt balance on quality of service and service‐level agreement violation. In this paper, a novel VMM algorithm based on Lion‐Whale optimization is developed by integrating the Lion optimization algorithm and the Whale optimization algorithm. The optimal virtual machine (VM) migration is performed by the Lion‐Whale VMM based on a new fitness function in the regulation of the resource use, migration cost, and energy consumption of VM placement. The experimentation of the proposed VM migration strategy is performed over 4 cloud setups with a different configuration which are simulated using CloudSim toolkit. The performance of the proposed method is validated over existing optimization‐based VMM algorithms, such as particle swarm optimization and genetic algorithm, using the performance measures, such as energy consumption, migration cost, and resource use. Simulation results reveal the fact that the proposed Lion‐Whale VMM effectively outperforms other existing approaches in optimal VM placement for cloud computing environment with reduced migration cost of 0.01, maximal resource use of 0.36, and minimal energy consumption of 0.09.     

一种粒子群算法与FCSS相结合的模糊球壳聚类算法

模糊球壳聚类算法（FCSS）广泛应用于模式识别与机器学习领域。由于其采用传统的基于梯度法和交替寻优策略求解模型,对初始值比较敏感,往往只能得到模型的局部极值点,从而影响聚类的效果,甚至使所采用的方法失效。本文将现代全局优化方法之一的粒子群优化算法与模糊球壳聚类算法（FCSS）相结合,利用粒子群算法良好的全局收敛能力来改善传统聚类算法易于陷入局部极值的缺陷,从而得到一种新的球壳聚类算法（PSO-FCSS）,数值实验表明,新方法对球壳形数据有令人满意的聚类效果。     

关于MapReduce程序运行时间的可靠估算

As data volume grows , many enterprises are considering using MapReduce for its simplicity. However, how to evaluate the performance improvement before deployment is still an issue. Current researches of MapReduce performance are mainly based on monitoring and simulation, and lack mathematical models . In this paper, we present a simple but powerful performance model for the prediction of the execution time of a MapReduce program with limited resources. We study each component of MapReduce framework, and analyze the relation between the overall performance and the number of mappers and reducers based on our model. Two typical MapReduce programs are evaluated in a small cluster with 13 nodes. Experimental results show that the mathematical performance model can estimate the execution time of MapReduce programs reliably. According to our model, number of mappers and reducers can be tuned to form a better execution pipeline and lead to better performance. The model also points out potential bottlenecks of the framework and future improvement.     

支持应急通信的认知网络QoS技术

文章提出支持应急通信的认知网络体系结构与模型,以及在此基础上涉及的认知模型服务质量控制和优化策略、智能决策和学习算法,目标在于通过对网络环境状态的观察和学习,智能决策并自适应调整节点行为,进而达到对网络性能的优化。     

OW‐SVM: Ontology and whale optimization‐based support vector machine for privacy‐preserved medical data classification in cloud

Cloud is a multitenant architecture that allows the cloud users to share the resources via servers and is used in various applications, including data classification. Data classification is a widely used data mining technique for big data analysis. It helps the learners to discover hidden data patterns by training massive data collected from the real world. Because this trained model is the private asset of an entity, it should be protected from all other noncollaborative entities. Therefore, it is essential to take effective measures to preserve the confidential data. The objective of this paper is to preserve the privacy of the confidential data in the cloud environment by introducing the medical data classification method. In view of that, this paper presents a method for medical data classification using a novel ontology and whale optimization‐based support vector machine (OW‐SVM) approach. Initially, privacy‐preserved data are developed adopting Kronecker product bat approach, and then, ontology is built for the feature selection process. Ontology and whale optimization‐based support vector machine is then proposed by integrating ontology and whale optimization algorithm into SVM, in which ontology and whale optimization algorithm is used for the feasible selection of kernel parameters. The experiment is done using 3 heart disease datasets, such as Cleveland, Switzerland, and Hungarian. In a comparative analysis, the performance of the OW‐SVM approach is compared with that of K‐nearest neighbor, Naive Bayes, decision tree, SVM, and OW‐SVM, using accuracy, sensitivity, specificity, and fitness, as the evaluation metrics. The OW‐SVM approach could achieve maximum performance with accuracy of 83.21%, the sensitivity of 91.49%, specificity of 73%, and fitness of 81.955, outperforming existing comparative techniques.     

Order@Cloud: An agnostic meta‐heuristic for VM provisioning,adaptation, and organisation

We propose a flexible meta‐heuristic framework for virtual machine (VM) organisation, provisioning, and adaptation in the cloud domain, based on migration costs and environment constraints. Order@Cloud improves VM placements according to multiple objectives represented by rules, qualifiers, and improvement cost, which can be easily modified and extended. Order@Cloud theoretically guarantees the adoption of a better set of placements, after considering their costs and benefits, by prioritising the worst VM placements. While existing solutions address only specific objectives, our framework is objective‐agnostic and extensible, which enables the adoption and implementation of new policies and priorities. We conduct experiments using a real cloud environment data and discuss the framework's performance, flexibility, and optimality and provide insights on the challenges and benefits of deploying this framework.     

分层异构信号处理平台调度方法研究

异构多处理器的高效性和可靠性能够满足日趋复杂的信号处理任务需求,因此分层异构系统已成为信号处理平台的发展趋势.为提高平台强实时性并解决高吞吐量的问题,文中对分层异构信号处理平台的软硬件模块及架构进行了研究,并采用有向无环图对组件任务及硬件资源进行建模.将已提出的调度算法按照任务类型、调度目标、调度过程和研究方法进行分类...     

A Unified Framework of the Cloud Computing Service Model

After a comprehensive literature review and analysis, a unified cloud computing framework is proposed, which comprises MapReduce, a vertual machine, Hadoop distributed file system (HDFS), Hbase, Hadoop, and virtualization. This study also compares Microsoft, Trend Micro, and the proposed unified cloud computing architecture to show that the proposed unified framework of the cloud computing service model is comprehensive and appropriate for the current complexities of businesses. The findings of this study can contribute to the knowledge for academics and practitioners to understand, assess, and analyze a cloud computing service application.     

Time slot assignment in a heterogeneous environment of a SS/TDMA system

In this paper we consider the time slot assignment for a heterogeneous environment in which circuit-switched traffic and packet-switched traffic share the same satellite. In constructing a single time division multiple access (TDMA) frame for both traffic types, their different characteristics must be taken into account. This problem is known to be NP-complete and a couple of heuristic partial optimization algorithms have been developed. In this paper, we first provide a theoretical result to improve the existing partial optimization algorithms; then a fully optimizing heuristic algorithm is presented. Simulation results show that our algorithm provides a much better solution quality than existing ones. © 1997 John Wiley & Sons, Ltd.