基于贝叶斯分类器的主题爬虫研究*

主题爬虫是实现定题搜索引擎的核心技术。提出了基于贝叶斯分类器实现主题爬虫的方法，介绍了基于贝叶斯分类器的主题爬虫的系统结构以及系统关键部分的实现，包括URL队列、爬行历史、页面下载以及页面分析，并重点介绍了基于贝叶斯分类器的主题相关度算法。爬虫使用改进的TF-IDF算法来提取网页内容的特征，并采用贝叶斯分类器计算其主题相关度。实验结果表明,在搜索大量网络资源的情况下，贝叶斯分类器比PageRank算法更适合用于实现主题爬虫。

Research on focused crawler based on Bayes classifier

Focused crawler is the core technology to implement a focused search engine. This paper made a research on focused crawler based on Bayes classifier, introduced the system architecture and key implementation of a focused crawler based on Bayes classifier. The key parts of the crawler include URL queue, crawling history, page downloader and page analyzer. This paper focused on the page ranking algorithm based on Bayes classifier.Used an improved TF-IDF algorithm to extract the characteristics of the page content, and adopted Bayes classifier to compute the page rank. The experiment results show that Bayes classifier is a better choice than PageRank algorithm to implement a focused crawler, under the condition of large network of resources.