基于新统计量mt的汉语自动分词方法研究

汉语自动分词是进行中文信息处理的基础。传统分词需要大规模加工过的熟语料库做为测试集来训练模型以获取参数,代价高昂。在互信息和t-测试差的基础上,通过将两者进行线性和非线性组合,提出了一个新的统计量mt。该统计量所需的所有统计数据直接从待切分的生语料中获得,无须大规模加工过的熟语料和人工干预,大大降低了分词成本。测试结果显示,该统计量关于字间位置的分词正确率为80.14%,比单独使用互信息和t-测试差分别提高了6.83%和7.27%。

Statistical-based Approach to Chinese Word Segmentation

Automatic Chinese word segmentation is the basis of Chinese information processing.It is expensive for obtaining parameter to use lage-scale annotated corpus.This paper presents a new statistical measure mt,in terms of a linear and non-linear combination of two common statistical measures,mutual information and difference of t-test.All statistical data needed derived from raw corpus automatically without using lage-scale annotated corpus,which aim to decrease cost.The experimental result shows segmentation accuracy reachs 80.14%.The segmentation is better compared to using mutual information and difference of t-test singly.