基于小句复合体的句子边界自动识别研究

自然语言文本的语法结构层次包括语素、词语、短语、小句、小句复合体、语篇等。其中,语素、词、短语等相关处理技术已经相对成熟,而句子的概念至今未有公认的、适用于语言信息处理的界定。该文重新审视了语言学中句子的定义和自然语言处理中句子的切分问题,提出了中文句子切分的任务;基于小句复合体理论将句子定义为最小的话头自足的标点句序列,也就是自足的话题结构,并设计和实现了基于BERT的边界识别模型。实验结果表明,该模型对句子边界自动识别正确率、F₁值分别达到88.37%、83.73%,识别效果优于按照不同的标点符号机械分割的效果。

Automatic Recognition of Sentence Boundary Based on Clause Complex

The grammatical structure of natural language text consists of words, phrases, sentences, clause complexes and texts. This paper re-examines the definition of sentences in linguistics and the segmentation of sentences in natural language processing, and puts forward the task of Chinese sentence segmentation. Based on the theory of clause complex, the sentence is defined as the smallest topic self-sufficient punctuation sequence, and a sentence boundary recognition model based on BERT is designed and implemented. The experimental results show that the accuracy and F₁ value of the model are 88.37% and 83.73%, respectively, much better than that of mechanical segmentation according to punctuation marks.