基于全局折扣的统计语言模型平滑技术

数据平滑是用来解决统计语言模型在实际应用中遇到的数据稀疏问题.现有平滑技术利用不同的折扣和补偿策略来处理数据稀疏问题,在计算复杂性与合理性方面各有其优缺点.针对二元模型,笔者提出了一种基于全局折扣GD(Global Discount)的平滑技术,其基本思想是对模型中每个二元对的频率值都进行不同程度的折扣,并用低阶模型对零概率事件进行补偿,通过极小化困惑度原则体现了模型的合理性.实验结果表明该平滑技术优于目前常用的Katz平滑技术.

Smoothing Technique for Statistical Language Model Based on Global Discount

Smoothing techniques are mainly used to solve the problem of sparse data for statistical language model. The present smoothing techniques deal with the data sparse problem using different discount and compensate strategy, and they have different merit or shortcoming on complexity and rationality. This paper presents a new kind of smoothing technique based on global discount for Bi-gram model. The model parameters, probabilities for bigram, are discounted according to frequency of bigram, and are compensated according to lower-level model for unseen events in the model, whose rationality is indicated by minimizing the perplexity. Experiment results show that the technique is superior to commonly used Katz smoothing technique.