统计模型在中文文本挖掘中的应用

本文讨论了中文文本挖掘的三个问题:分词、关键词提取和文本分类。对分词问题,介绍了基于层叠隐马尔可夫模型的ICTCLAS分词法,以及将词与词之间的分隔视为缺失数据并用EM算法求解的WDM方法;对关键词提取问题,提出了贝叶斯因子法,并介绍了使用稀疏回归的CCS方法;对文本分类问题,介绍了根据关键词频率建立分类器的方法,以及先建立主题模型再根据主题概率建立分类器的方法。本文通过两组文本数据对上述方法进行比较,并给出使用建议。

Applications of Statistical Models in Chinese Text Mining

This paper discusses three problems in Chinese text mining,including word segmentation,keyword extraction and text classification.For the word segmentation problem,we introduce the ICTCLAS method that is based on a hierarchical hidden Markov model,and the WDM method that treats the segmentation between words as missing data and uses the EM algorithm to find the solution.For the keyword extraction problem,we propose a method based on Bayes Factor,and introduce the CCS method that uses sparse regression.For the text classification problem,we introduce a method that builds classifiers on keyword frequencies,and another method that first trains topic models and then builds classifiers on topic proportions.This paper then compares the above methods using two text datasets,and offers suggestions on their practical use.