| 节点对等WebSpider设计与实现 |
| |
| 引用本文： | 张林才,张燕,王红霞.节点对等WebSpider设计与实现[J].微机发展,2010(3):195-198,202. |
| |
| 作者姓名： | 张林才 张燕 王红霞 |
| |
| 作者单位： | 辽宁石油化工大学计算机与通信工程学院;北京青年政治学院计算机系; |
| |
| 基金项目： | 辽宁省自然科学基金(20052211) |
| |
| 摘 要： | 由于互联网具有海量信息并且快速增长，提高搜索引擎的信息采集器WebSpider的数据采集和更新速度有重要意义。受计算资源限制，单机多线程WebSpider的采集速率不高。带中心节点的分布式并行webSpider又容易产生中心节点瓶颈问题。利用ProActive网格网络并行分布计算中间件提供的主动对象技术、网络并行计算技术、自动部署机制等设计和实现了一个名为P—Spider2．0的节点对等的分布式并行WebSpider，并设计了一个基于Raibin算法的URL去重算法。实验表明该WebSpider方便管理和部署，并且比单机多线程webSpider具有更高的采集速率。
|
| 关 键 词： | 网络爬虫 ProActive 并行 分布式 节点对等 |
Design and Realization of Peer-to-Peer Web Spider |
| |
| ZHANG Lin-cai,ZHANG Yan,WANG Hong-xia.Design and Realization of Peer-to-Peer Web Spider[J].Microcomputer Development,2010(3):195-198,202. |
| |
| Authors: | ZHANG Lin-cai ZHANG Yan WANG Hong-xia |
| |
| Affiliation: | ZHANG Lin-cai1,ZHANG Yan1,WANG Hong-xia2 (1.School of Computer , Communication Engineering,Liaoning Shihua University,Fushun 113001,China,2.Computer Science Dept.,Beijing Young , Political College,Beijing 100102,China) |
| |
| Abstract: | With the rapid growth of Internet's massive information,it becomes very significant to enhance the data acquisition and update speed of search engines' information collector:Web Spider.Bottleneck caused by center node is apt to occur in the distributed parallel Web Spider with center node.The peer-to-peer distributed parallel Web Spider named P-Spider2.0 is designed and realized with the active object,network parallel computing technology and automatic deployment of ProActive which is a grid and network par... |
| |
| Keywords: | Web Spider ProActive parallel distributed peer-topeer |
| 本文献已被 CNKI 维普 等数据库收录！ |
