隐含语义索引及其在中文文本处理中的应用研究

信息检索本质上是语义检索,而传统信息检索系统都是基于独立词索引,因此检索效果并不理想,隐含语义索引是一种新型的信息检索模型,它通过奇异值分析,将词向量和文档向量投影到一个低维空间,消减了词和文档之间的语义模糊度,使得文档之间的语义关系更为明晰。实验和理论结果证实了隐含语义索引能够取得更好的检索效果。本文论述了隐含语义索引的理论基础,研究了隐含语义索引在中文文本处理中的应用,包括中文文本检索、中文文本分类和中文文本聚类等。     

基于模糊语言方法的信息检索系统的研究

该文提出了一个基于模糊语言方法的信息检索系统模型。该系统分为查询界面子系统、数据库子系统和检索子系统三大部分。在查询界面子系统,用布尔表达式表示用户的查询请求,并对每个查询关键词赋予了两种不同语义的语言值权重,该权重表达了用户的模糊检索要求;在数据库子系统,用索引词一文档模糊矩阵表示待检索的文档,对每个索引词。根据其在文档中的出现频率大小。引入了数值权重;在检索子系统,运用模糊语言方法,对用户输入的布尔查询表达式与索引词一文档模糊矩阵进行自底向上的模糊匹配,最后返回满足用户要求的检索结果。相对于传统的基于查询关键词精确匹配的检索系统而言,该系统能较好地满足用户查询要求中的灵活性。     

针对中文检索的Lucene改进策略

为了提高基于Lucene中文检索系统的检索精度和效率,通过分析Lucene的结构,在系统中加入了中文分词模块和索引文档预处理模块。给出了具体的实验方法和实验过程,对改进原理和实验数据进行了分析,表明了加入中文分词模块和在索引预处理模块中采用提取特定数量的特征词来替代文档的方法能够有效提高Lucene检索系统的效率和精度,增强Lucene检索系统中文的性能。     

概率潜在语义检索模型中中文索引技术的比较研究

概率潜在语义检索模型使用统计的方法建立“文档—潜在语义一词”之间概率分布关系并利用这种关系进行检索。本文比较了在概率潜在语义检索模型中不同中文索引技术对检索效果的影响，考察了基于分词、二元和关键词抽取三种不同的索引技术，并和向量空间模型作了对比分析。实验结果表明：在概率潜在语义检索模型中，词的正确切分能提高检索的平均精度。     

基于Lucene的中文全文检索系统的研究与设计

提出了一种基于Lucene的中文全文检索系统模型.通过分析Lucene的系统结构,系统采用了基于统计的网页正文提取技术,并且加入了中文分词模块和索引文档预处理模块来提高检索系统的效率和精度.在检索结果的处理上,采用文本聚类的办法,使检索结果分类显示,提高了用户的查找的效率.实验数据表明,该系统在检索中文网页时,在效率,精度和结果处理等方面性能明显提高.     

一种基于领域本体的论文检索方法的研究与应用

针对传统的论文检索方法缺乏语义理解,检索结果相关度不高的缺点,采用基于语义网络的文档语义表达模型,提出一种基于领域本体的检索方法。首先结合学科分类体系构建领域本体,并对论文文档进行语义索引;然后根据本体知识和索引信息构建基于语义网络的文档语义表达模型;最后改进用户查询与语义网络的相关度算法,综合关键词和语义的方法实现结果排序。实验结果表明,该方法能有效地提高论文检索的准确率和召回率。     

一种结合超链接分析的搜索引擎排序方法

为了提高搜索引擎的检索性能，文章设计了一种搜索引擎的综合排序方法。它采用改进的布尔检索模式、中文分词、超链接分析以及索引链接文本等技术，主要具有以下特点：对经典布尔型检索模式所作的改进使得文档相关度不再是严格的0或1；超链接分析通过互联网的链接结构计算出每个网络文档的质量；通过中文分词和索引链接文本可以更加准确地获得一个网络文档的信息内涵。将3者结合可以充分利用各自优势而弥补不足。     

微博中基于增强型倒排索引的特定文档影响力估计算法

微博搜索系统中，将微博帖子根据搜索相关性和重要性进行排序，并通过列表的方式返回结果，是目前信息内容的主要展示手段。基于向量空间模型的打分函数被广泛地应用于该类系统中。事实上，微博系统中的帖子重要性打分函数实际取值并不为用户所见，文档的影响力通过排名的方式表现出来。对于一个检索外的文档，如何衡量其在信息检索系统文库中的影响力？一般搜索引擎或信息检索系统并不能很好地回答该问题。在微博短文本的基础上引入了社交影响力这一概念，并通过在文本倒排索引基础上设置反向位置标记，给出了一种全新的影响力度量指标，有效地回答了前述问题。理论分析和数据实验验证了算法的有效性和效率。     

基于文档实例的中文信息检索

传统的信息检索系统基于关键词建立索引并进行信息检索.这些系统存在查询返回文档集大、准确率低和普通用户不便于构造查询等不足.为此,该文提出基于文档实例的信息检索,即以已有文档作为样本,在文档库中检索与样本文档相似的所有文档.文中给出了基于文档实例的中文信息检索的解决方法和实现技术.初步实验结果表明该方法是行之有效的.     

基于相关术语群的中文查询扩展

查询词语和文档中词语的不匹配是影响文本信息检索效果的一个关键因素.查询扩展技术可以在一定程度上解决这种词的不匹配问题,然而,实验表明,通常简单的查询扩展并不能稳定地提高中文信息检索的检索精度.利用自动构建的相关术语群来进行查询扩展以提高中文检索的效果.在NTCIR中文信息检索测试集上进行的实验表明,相对于传统的查询扩展方法,在检索效果上取得了平均24.5%的提高.     

结合邻近度的语义位置语言检索模型

在传统的检索模型中,文档与查询的匹配计算主要考虑词项的统计特征,如词频、逆文档频率和文档长度,近年来的研究表明应用查询词项匹配在文档中的位置信息可以提高查询结果的准确性。如何更好地刻画查询词在文档中的位置信息并建模,是研究提高检索效果的问题之一。该文在结合语义的位置语言模型(SPLM)的基础上进一步考虑了词的邻近信息,并给出了用狄利克雷先验分布来计算邻近度的平滑策略,提出了结合邻近度的位置语言检索模型。在标准数据上的实验结果表明,提出的检索模型在性能上要优于结合语义的位置语言模型。     

基于概率潜在语义分析的中文信息检索

传统的信息检索模型把词看作孤立的单元,没有考虑自然语言中存在大量的同义词、多义词现象,对召回率和准确率有不利的影响。概率潜在语义模型使用统计的方法建立“文档-潜在语义-词”之间概率分布关系并利用这种关系进行检索。该文将概率潜在语义模型用于中文信息检索,实验结果表明,概率潜在语义模型相对于传统的向量空间模型能够显著地提高检索的平均精度。     

Chinese word segmentation as morpheme-based lexical chunking

Chinese word segmentation plays an important role in many Chinese language processing tasks such as information retrieval and text mining. Recent research in Chinese word segmentation focuses on tagging approaches with either characters or words as tagging units. In this paper we present a morpheme-based chunking approach and implement it in a two-stage system. It consists of two main components, namely a morpheme segmentation component to segment an input sentence to a sequence of morphemes based on morpheme-formation models and bigram language models, and a lexical chunking component to label each segmented morpheme’s position in a word of a special type with the aid of lexicalized hidden Markov models. To facilitate these tasks, a statistically-based technique is also developed for automatically compiling a morpheme dictionary from a segmented or tagged corpus. To evaluate this approach, we conduct a closed test and an open test using the 2005 SIGHAN Bakeoff data. Our system demonstrates state-of-the-art performance on different test sets, showing the benefits of choosing morphemes as tagging units. Furthermore, the open test results indicate significant performance enhancement using lexicalization and part-of-speech features.     

Negations and document length in logical retrieval

An unsolved problem in logic-based information retrieval is how to obtain automatically logical representations for documents and queries. This problem limits the impact of logical models for information retrieval because their full expressive power cannot be harnessed. In this paper we propose a method for producing logical document representations which goes further than other simplistic “bag-of-words” approaches. The suggested procedure adopts popular information retrieval heuristics, such as document length corrections and global term distribution. This work includes a report of several experiments applying partial document representations in the context of a propositional model of information retrieval. The benefits of this expressive framework, powered by the new logical indexing approach, become apparent in the evaluation.     

A new full-text indexing model with low space overhead for chinese text retrieval

Text retrieval systems require an index to allow efficient retrieval of documents at the cost of some storage overhead. This paper proposes a novel full-text indexing model for Chinese text retrieval based on the concept of adjacency matrix of directed graph. Using this indexing model, on one hand, retrieval systems need to keep only the indexing data, instead of the indexing data and the original text data as the traditional retrieval systems always do. On the other hand, occurrences of index term are identified by labels of the so-called s-strings where the index term appears, rather than by its positions as in traditional indexing models. Consequently, system space cost as a whole can be reduced drastically while retrieval efficiency is maintained satisfactory. Experiments over several real-world Chinese text collections are carried out to demonstrate the effectiveness and efficiency of this model. In addition to Chinese, The proposed indexing model is also effective and efficient for text retrieval of other Oriental languages, such as Japanese and Korean. It is especially useful for digital library application areas where storage resource is very limited (e.g., e-books and CD-based text retrieval systems).     

Using semantic components to search for domain-specific documents: An evaluation from the system perspective and the user perspective

We seek to leverage an expert user's knowledge about how information is organized in a domain and how information is presented in typical documents within a particular domain-specific collection, to effectively and efficiently meet the expert's targeted information needs. We have developed the semantic components model to describe important semantic content within documents. The semantic components model for a given collection (based on a general understanding of the type of information needs expected) consists of a set of document classes, where each class has an associated set of semantic components. Each semantic component instance consists of segments of text about a particular aspect of the main topic of the document and may not correspond to structural elements in the document. The semantic components model represents document content in a manner that is complementary to full text and keyword indexing. This paper describes how the semantic components model can be used to improve an information retrieval system. We present experimental evidence from a large interactive searching study that compared the use of semantic components in a system with full text and keyword indexing, where we extended the query language to allow users to search using semantic components, to a base system that did not have semantic components. We evaluate the systems from a system perspective, where semantic components were shown to improve document ranking for precision-oriented searches, and from a user perspective. We also evaluate the systems from a session-based perspective, evaluating not only the results of individual queries but also the results of multiple queries during a single interactive query session.     

A constraint-based approach to shape management in multimedia databases

Shape management is an important functionality in multimedia databases. Shape information can be used in both image acquisition and image retrieval. Several approaches have been proposed to deal with shape representation and matching. Among them, the data-driven approach supports searches for shapes based on indexing techniques. Unfortunately, efficient data-driven approaches are often defined only for specific types of shape. This is not sufficient in contexts in which arbitrary shapes should be represented. Constraint databases use mathematical theories to finitely represent infinite sets of relational tuples. They have been proved to be very useful in modeling spatial objects. In this paper, we apply constraint-based data models to the problem of shape management in multimedia databases. We first present the constraint model and some constraint languages. Then, we show how constraints can be used to model general shapes. The use of a constraint language as an internal specification and execution language for querying shapes is also discussed. Finally, we show how a constraint database system can be used to efficiently retrieve shapes, retaining the advantages of the already defined approaches.     

Document ranking and the vector-space model

Efficient and effective text retrieval techniques are critical in managing the increasing amount of textual information available in electronic form. Yet text retrieval is a daunting task because it is difficult to extract the semantics of natural language texts. Many problems must be resolved before natural language processing techniques can be effectively applied to a large collection of texts. Most existing text retrieval techniques rely on indexing keywords. Unfortunately, keywords or index terms alone cannot adequately capture the document contents, resulting in poor retrieval performance. Yet keyword indexing is widely used in commercial systems because it is still the most viable way by far to process large amounts of text. Using several simplifications of the vector-space model for text retrieval queries, the authors seek the optimal balance between processing efficiency and retrieval effectiveness as expressed in relevant document rankings     

Multi-Scale Spoken Document Retrieval for Cantonese Broadcast News

This paper presents the application of a multi-scale paradigm to Chinese spoken document retrieval (SDR) for improving retrieval performance. Multi-scale refers to the use of both words and subwords for retrieval. Words are basic units in a language that carry lexical meaning, and subword units (such as phonemes, syllables or characters) are building components for words. Retrieval using subword indexing units is better than retrieval using words because of the robustness of subword units to out-of-vocabulary (OOV) words during speech recognition and ambiguities in word segmentation. Experimental results have demonstrated that subword bigrams can bring improvement in retrieval performance over words (～9.56%). Application of multi-scale fusion to SDR aims at combining the lexical information of words and the robustness of subwords. This work presents the first detailed investigation for a Cantonese broadcast news retrieval task using two different multi-scale fusion approaches: pre-retrieval fusion and post-retrieval fusion. Multi-scale retrieval using both words and syllable bigrams achieves improvement in retrieval performance (～1.90%) over retrieval on the composite scales.