基于生物计算的分布式计算系统

分布式计算技术提供了充分利用现有网络资源的有效途径。该文论述了基于解决生物计算中难解问题的具有开放接口的分布式并行计算系统的设计与实现技术。系统兼有开放式、异构性、容错性与易用性等特点。讨论了系统的容错性机制、检查点策略及任务调度算法。对Motif Finding问题的求解验证表明,分布式并行计算机制能大大缩短问题的求解时间,为计算领域的难解问题提供有效的解决途径。     

多Agent 主从粒子群分布式计算框架

面向大规模复杂优化问题,提出了一个基于并行粒子群优化的分布式Agent计算框架.框架中使用一个主群(master swarm)来演化问题的完整解,并使用一组从群(slave swarm)来并行优化一组子问题的解,主群和从群通过交替执行来提高问题的求解效率.采用异步组结构,主群/从群中的各类Agent共享一个解群,并通过相互协作,对解群进行构造、改进、修补、分解和合并等演化操作.该框架可用于求解复杂的约束多目标优化问题.通过一类典型运输问题上的实验,其结果表明,所提出的方法明显优于另外两种先进的演化算法.     

Framework for Task Scheduling in Heterogeneous Distributed Computing Using Genetic Algorithms

An algorithm has been developed to dynamically schedule heterogeneous tasks on heterogeneous processors in a distributed system. The scheduler operates in an environment with dynamically changing resources and adapts to variable system resources. It operates in a batch fashion and utilises a genetic algorithm to minimise the total execution time. We have compared our scheduler to six other schedulers, three batch-mode and three immediate-mode schedulers. Experiments show that the algorithm outperforms each of the others and can achieve near optimal efficiency, with up to 100,000 tasks being scheduled     

新型分布式计算系统中的异构任务调度框架

随着大数据和机器学习的火热发展,面向机器学习的分布式大数据计算引擎随之兴起.这些系统既可以支持批量的分布式学习,也可以支持流式的增量学习和验证,具有低延迟、高性能的特点.然而,当前的一些主流系统采用了随机的任务调度策略,忽略了节点的性能差异,因此容易导致负载不均和性能下降.同时,对于某些任务,如果资源要求不满足,则会导...     

A Riemannian Framework for Tensor Computing

Tensors are nowadays a common source of geometric information. In this paper, we propose to endow the tensor space with an affine-invariant Riemannian metric. We demonstrate that it leads to strong theoretical properties: the cone of positive definite symmetric matrices is replaced by a regular and complete manifold without boundaries (null eigenvalues are at the infinity), the geodesic between two tensors and the mean of a set of tensors are uniquely defined, etc. We have previously shown that the Riemannian metric provides a powerful framework for generalizing statistics to manifolds. In this paper, we show that it is also possible to generalize to tensor fields many important geometric data processing algorithms such as interpolation, filtering, diffusion and restoration of missing data. For instance, most interpolation and Gaussian filtering schemes can be tackled efficiently through a weighted mean computation. Linear and anisotropic diffusion schemes can be adapted to our Riemannian framework, through partial differential evolution equations, provided that the metric of the tensor space is taken into account. For that purpose, we provide intrinsic numerical schemes to compute the gradient and Laplace-Beltrami operators. Finally, to enforce the fidelity to the data (either sparsely distributed tensors or complete tensors fields) we propose least-squares criteria based on our invariant Riemannian distance which are particularly simple and efficient to solve.     

利用分布式运算实现DNA序列间物理特性的比较

在DNA序列相关物理特性的模拟计算中,计算时间随DNA比较序列数的增加而延长。该研究成功应用分布式运算实现了对DNA序列间相关物理特性的模拟计算。该研究表明:对较大的DNA序列进行比较时,使用JavaParty和增加线程数能有效地缩短运行时间。     

基于张量的取向估计方法的理论框架

取向估计的主要目的是计算出图像等多维信号各点的取向信息,在图像处理和机器视觉的底层处理中具有广泛的应用.在总结现有基于张量的取向估计方法的基础上,文中提出了基于张量的取向估计方法的理论框架,并从取向张量的构造这一核心问题入手证明各种基于张量的取向估计方法都可以统一到这一理论框架之下,从而有利于对这类方法进行深入研究或设...     

Gauss: A Framework for Verifying Scientific Computing Software

High performance scientific computing software is of critical international importance as it supports scientific explorations and engineering. Software development in this area is highly challenging owing to the use of parallel/distributed programming methods and complex communication and synchronization libraries. There is very little use of formal methods to debug software in this area, given that the scientific computing community and the formal methods community have not traditionally worked together. The Utah Gauss project combines expertise from scientific computing and formal methods in addressing this problem. We currently focus on MPI programs which are the kind that run on over 60% of world's supercomputers. These are programs written in C / C++ / FORTRAN employing message passing concurrency supported by the Message Passing Interface (MPI) library. Large-scale MPI programs also employ shared memory threads to manage concurrency within smaller task sub-groups, capitalizing on the recent availability of small-scale (e.g. single-chip) shared memory multiprocessors; such mixed programming styles can result in additional bugs. MPI libraries themselves can be buggy as they strive to implement complex requirements employing aggressive techniques such as multi-threading. We have built a model extractor that extracts from MPI C programs a formal model consisting of communicating processes represented in Microsoft's Zing modeling language. MPI library functions are also being modeled in Zing. This allows us to run formal analysis on the models to detect bugs in the MPI programs being analyzed. Our preliminary results and future plans are described; in addition, our contribution is to expose the special needs of this area and suggest specific avenues for problem- driven advances in software model-checking applied to scientific computing software development and verification.     

JDCS:实现高性能计算的分布式计算系统

分布对象计算技术提供了充分利用现有网络资源的有效途径。JavaRMI是当前比较成熟的一种分布对象技术,它提供了使用Java对象的简单和直接的方法。该文建立基于JavaRMI方法的适用于高性能计算的分布式计算系统JDCS。在JDCS中由网络上的计算结点构成服务器池,为用户提供高性能的计算服务。实现结果表明该系统可以获得较高的加速比。     

MapReduce框架下结合分布式编码计算的容错算法

随着分布式系统规模扩大及计算复杂度增加,分布式计算的平均故障修复时间和容错计算所产生的通信开销呈现日益上升趋势.结合分布式编码计算和副本冗余技术,提出一种新的容错算法.map节点应用分布式编码计算的思想,将数据冗余分配至多个计算节点创建编码中间结果,降低计算节点在shuffle阶段的数据传输量.reduce节点通过对接...     

一种分布式网格计算框架以及在大规模分子动力学模拟中的应用

分子动力学（molecular dynamics）模拟蛋白质等大分子内原子间的相互作用,蛋白质折叠所需的时间通常在微秒（10^-6s）量级，而进行模拟的时间步长在飞秒（10^-15s）量级，并且每步需要计算大量的相互作用（O（n^2），n为原子数），以致于无法模拟足够长时间的折叠过程．现今在满足精确度的需求下没有更好的模拟算法．最近，生物学家研究了一种分布式的动力学方法，使得可以利用分布在Internet上的计算机进行并行模批成为可能,本文的目标是设计并实现在分布式P2P和网格计算环境等多种异构计算资源下进行动力学模拟的可靠框架，以便更大限度地利用计算资源，加快计算过程．我们基于Java和web service技术，已经实现了对应用透明的计算框架，并已将它扩展到我们的网格计算环境,实验表明分子动力学模拟程序在该框架下运行良好．     

Hadoop分布式架构下大数据集的并行挖掘

基于Hadoop分布式计算平台,给出一种适用于大数据集的并行挖掘算法。该算法对非结构化的原始大数据集以及中间结果文件进行垂直划分以确保能够获得完整的频繁项集,将各个垂直分块数据分配给不同的Hadoop计算节点进行处理,以减少各个计算节点的存储数据,进而减少各个计算节点执行交集操作的次数,提高并行挖掘效率。实验结果表明,给出的并行挖掘算法解决了大数据集挖掘过程中产生的大量数据通信、中间数据以及执行大量交集操作的问题,算法高效、可扩展。     

一种分布式吴方法计算模型

吴方法是由我国科学家吴文俊院士开创的一个新兴研究领域.考虑到吴方法"分而治之"的思想非常适合分布式计算,将分布式计算技术引入到该方法的计算过程中,给出一种既可以在集群环境下,也可以在网格环境下实现的分布式吴方法计算框架.首先分析了吴方法分布式计算需求,并以特征列计算为例来说明吴方法分布式计算算法,然后讨论了符号计算基本数据类型:大整数和多项式的消息传递方法,最后简单给出了在网格环境下基于符号计算软件系统ELIMINO和网格中件间Globus Toolkits 3的分布式吴方法计算环境的设计、实现与实验结果.     

Cell: A Distributed Computing Modularization Concept

This paper presents a new language construct for distributed computing. This construct, called cell, allows one to simulate a variety of language constructs, Its salient features provide the programmer with: 1) an effective communication and synchronization scheme, 2) a mechanism to control the order in which various activities within a cell should be executed. We demonstrate the usefulness of our concepts by providing solutions to a variety of programming exercises.     

Bigflow: A General Optimization Layer for Distributed Computing Frameworks

As data volumes grow rapidly, distributed computations are widely employed in data-centers to provide cheap and efficient methods to process large-scale parallel datasets. Various computation models have been proposed to improve the abstraction of distributed datasets and hide the details of parallelism. However, most of them follow the single-layer partitioning method, which limits developers to express a multi-level partitioning operation succinctly. To overcome the problem, we present the NDD (Nested Distributed Dataset) data model. It is a more compact and expressive extension of Spark RDD (Resilient Distributed Dataset), in order to remove the burden on developers to manually write the logic for multi-level partitioning cases. Base on the NDD model, we develop an open-source framework called Bigflow, which serves as an optimization layer over computation engines from most widely used processing frameworks. With the help of Bigflow, some advanced optimization techniques, which may only be applied by experienced programmers manually, are enabled automatically in a distributed data processing job. Currently, Bigflow is processing about 3 PB data volumes daily in the data-centers of Baidu. According to customer experience, it can significantly save code length and improve performance over the intuitive programming style.     

使用内存缓存的迭代应用编程框架

迭代式计算是一类重要的大数据分析应用.在分布式计算框架MapReduce上实现迭代计算时,计算会被分解成多个作业并按作业依存关系顺序运行,这使得程序与分布式文件系统(DFS)有多次交互而影响程序执行时间.对这些交互相关数据的缓存会降低与DFS的交互时间,进而提升程序总体的性能.考虑到集群中的大量内存在多数情况下会处于空闲状态,提出了一种使用内存缓存的迭代式应用编程框架MemLoop.该系统从作业提交API、调度算法、缓存管理模块实现缓存管理以充分利用内存缓存迭代间可驻留数据与迭代内依存数据.我们将此框架与已有相关框架进行了比较,实验结果表明该框架能够提升迭代程序的性能.     

一个面向分布式程序的测试系统框架

提出了一个面向分布式程序的测试系统框架TFDS(test system framework for distributed software system),并介绍了它在异构网络中的一个实现原型PSET*(distributed progra m structure and event trace, revised version).框架的主要功能是对分布式程序进行单 元测试和集成测试.包括面向规约设计和源码分析的静态部分和面向程序执行和事件序列分 析的动态部分.在构件的基础上,PSET*的功能可以较容易