基于结构驱动的网络论坛采集路径研究*

网络论坛中蕴涵着大量具有实用价值和商业价值的信息，是搜索引擎和问答系统信息的重要来源。针对论坛结构复杂、链接种类繁多，以及容易陷入采集陷阱等问题，提出了一种基于结构驱动的采集路径选择方法。首先根据用户标注的少量类型数据，利用DOM树对采样网页基于网页结构进行结构聚类；其次根据各节点的评价进行采集路径选择；最后对翻页链接进行有效的识别和处理。实验表明，该方法采集的覆盖率和有效率明显优于传统算法，并且应用在中国科学院计算所舆情监测平台上取得了良好的效果。

Structure-driven based traversal strategy for Web forum crawling

Forums contain much practical and business information, which is the important source of information for search engines and question answering system. Complex structure of the forums, a great variety of links and the issues that being easy to fall into the trap of crawling are all the problems when collect information. This paper proposed a crawling method based on structure-driven path selection to solve these problems. First, used a small number of types of data marked by the users, and used DOM tree to cluster by structure based on Web-based structure. And then, chose the collected route according to the evaluation of each node, at last identified and processed the link to the page effectively. Experiments show that the coverage and efficiency of collection is better than the traditional algorithm. And get good results through the golaxy public opinion monitoring system of ICT.