EntropyRank: 基于主题熵的关键短语提取算法

关键短语提取是自然语言处理领域的一个重要子任务,其目的是自动识别出文本中的重要短语,现有方法主要强调词语间相关关系和词语自身影响力会影响关键短语提取效果。考虑到关键短语应准确地表示文档主题这一特点,该文提出一种基于主题熵的关键短语提取算法。该算法利用隐含狄利克雷分布训练文档和词的主题分布,并结合两个主题分布来表示特定文档下的词主题分布,然后计算词主题分布的信息熵即主题熵来表示词语自身影响力,最后在词共现网络上使用随机游走方法计算每个候选短语的得分。在6个公开数据集上的实验结果表明,与现有的无监督关键短语提取算法相比,该算法在F₁指标上能提高2.61%～6.98%。

EntropyRank: Keyphrase Extraction Algorithm Based on Topic Entropy

Key-phrase extraction aims to automatically identify important key-phrases from documents. Most existing methods are focused on the words' importance and the relation between words. Considering that key-phrase should closely related to the article's topics, we proposed an improved method based on topic entropy. Our work firstly use Latent Dirichlet Allocation to train the theme distribution of documents and words, and combine them to get the words' topic distribution of a specific document. Then words' topic entropy are worked out to represent the words' importance. Finally, we use random walk on words' co-occurrence graph to calculate the score of each candidate phrase. Experimental results show that proposed method has an improvement of 2.61%-6.98% in F1 score compared with the existing methods.