归纳学习与规则结合的分词方法的有效性考察

随着互联网的普及和网上电子文本信息的爆炸式的增加,自然语言处理技术面向动态的、变化的文本显得越来越必要.针对无切分语言的分词处理的主要难点是切分歧义和未知词的处理.基于归纳学习的分词方法,仅利用文本的表层信息,因此具有完全不依赖于某特定语言的优点.通过引入包含上下文信息的消歧处理规则,对基于归纳学习的分词方法进行改进.以归纳学习方法对未知词进行推测,抽出的规则用于歧义切分的消歧处理,提高了对切分歧义的处理精度.通过实验对规则的有效性进行了考察,并给出了改进方法的分词效果.

Evaluation of Word Segmentation Method Based on Inductive Learning and Rules

With the development of the Internet and increasing of on-line electronic text,it is necessary that natural language processing technology could deal with those dynamic,open texts.The difficult problem of word segmentation is processing of segmentation ambiguity and identifying of unknown words.The method based on Inductive Learning use only surface information of a text,so that it has an advantage that is entirely not dependent on any specific language.The Inductive Learning method is improved by using segmentation rules that contain information of context.The method predicts unknown words with Inductive Learning and process segment ambiguity by rules,segmentation rate is improved,usefulness of rules is evaluated and results of word segmentation are demonstrated.