基于DRPKP算法的文本去重研究与应用

SimHash算法是目前主流的文本去重算法,但它对于特定行业的文本数据在主题方面的天然相似性特点并没有特殊的考虑.基于多年在金融证券行业信息管理和数据整合的经验,本文分析目前文本去重方法存在的问题,特别针对SimHash算法在特定行业文本去重中的不足,创新地提出一种基于段落主题的文本去重方法（简称DRPKP算法）,通过对去重准确率、覆盖率和去重时间3个指标进行对比测试,DRPKP算法比SimHash算法准确率可提高24.5％、覆盖率可提高16.34％,且去重时间更短.

Research and Application on Text Duplication Removal Based on DRPKP Algorithm

SimHash algorithm is one of the best algorithm for text duplication detection and removal.However,it has less consideration on the naturalsimilarity of text in specific fields.Based on our experience in information management and data integration in financing and securities industry,we analyzemost text duplication removal algorithms today,especially focus onSimHash algorithm,and propose an newalgorithm for text duplication detection and removal which is based on paragraph key phrase（DRPKP）.We appliedour algorithm to detect and remove text duplication in real data set onGuo Tai Jun An＇s Financial Information and Unified Information Retrieval Platform.In comparison withSimHash algorithm,our DRPKPalgorithm performs better with the precision ofduplication removal increased by 24.5％,andthe recallincreased by 16.34％; meanwhile,our DRPKPalgorithm also shows an advantage in operating time.