基于遗传算法的主题爬虫

针对目前主题网络爬虫搜索策略难以在全局范围内找到最优解，通过对遗传算法的分析与研究，文中设计了一个基于遗传算法的主题爬虫方案。引入了结合文本内容的PageRank算法；采用向量空间模型算法计算网页主题相关度；采取网页链接结构与主题相关度来评判网页的重要性；依据网页重要性选择爬行中的遗传因子；设置适应度函数筛选与主题相关的网页。与普通的主题爬虫比较，该策略能够获取大量主题相关度高的网页信息，能够提高获取的网页的重要性，能够满足用户对所需主题网页的检索需求，并在一定程度上解决了上述问题。

Focused Crawling Based on Genetic Algorithms

Optimized solution can＇t be found in the global scope based on the present searching strategy of focused crawler. A focused crawler method based on genetic algorithm is proposed through the analysis and study of genetic algorithm. This method introduces the PageRank algorithm combined with text contents, computes the page topic similarity with vector space model algorithm, and judges the importance of web page according to web link structure and topic similarity. At the same time, the genetic factors are selected on basis of the importance of web page. The system sets fitness function to select pages relevant with topic. Compared to focused crawler, the topic crawler based on genetic algorithms could obtain the web pages which have strong correlation with subjects, and improve the impor- tance of access web pages, and satisfy user＇ s demand for searching topic webs they,re interested in. So in a certain extent, the above problems are solved.