基于倒排索引的可搜索加密数据共享方案

针对当前医疗数据共享难、数据被攻击、密文搜索效率低的问题，提出了一种基于倒排索引的可搜索加密数据共享方案。采用基于私有链和联盟链的双链结构存储数据并实现数据共享；设计了一种新型倒排索引结构，防止敏感数据被攻击；提出了一种基于新型倒排索引结构的密文搜索算法，利用可搜索加密技术将陷门信息提交至倒排索引结构中实现密文搜索。实验表明，所提出的方案可以有效保证数据安全，提高搜索效率。     

搜索引擎分块索引技术研究

搜索引擎通常使用签名文件、倒排索引、后缀树/后缀数组来构建索引,其中倒排文件由于性能可靠、实现简单,得到广泛应用.但是在现今海量数据情况下,有时倒排索引仍显力不从心,导致检索性能不佳,提出了分块索引术,可以有效提高基于倒排索引搜索系统检索效率.     

一种实时更新索引结构的设计与实现

在搜索引擎的信息检索中,索引性能的优劣是影响检索质量的一个重要因素.本文针对面向主题搜索引擎内容覆盖范围窄、更新速度快的特点,设计了一种实时更新的索引结构,该方案的倒排索引结构打破了传统索引单一结构的形式,由主倒排索引、附加倒排索引和删除文件列表组成,很好的解决了索引的更新问题,实验结果显示该索引结构具有良好的性能.     

基于策略爬行与混合索引的医药行业垂直搜索引擎的设计与实现

随着互联网应用的深入,越来越多的用户希望通过搜索引擎获得特定行业的相关信息,通用搜索引擎无法有效地满足相应需求。文中主要介绍医药行业垂直搜索引擎的设计与实现。设计基于智能搜索引擎的架构,采用了任务驱动的聚焦搜索、隐藏搜索技术;字词混合倒排索引及优化的字倒排索引、检索技术。提供了资源收集阶段的可控策略爬行,和高效的索引、检索功能。实现了针对医药行业的高专业度、高准确率、高效率的信息垂直搜索。     

基于策略爬行与混合索引的医药行业垂直搜索引擎的设计与实现

随着互联网应用的深入,越来越多的用户希望通过搜索引擎获得特定行业的相关信息,通用搜索引擎无法有效地满足相应需求。文中主要介绍医药行业垂直搜索引擎的设计与实现。设计基于智能搜索引擎的架构,采用了任务驱动的聚焦搜索、隐藏搜索技术;字词混合倒排索引及优化的字倒排索引、检索技术。提供了资源收集阶段的可控策略爬行,和高效的索引、检索功能。实现了针对医药行业的高专业度、高准确率、高效率的信息垂直搜索。     

一种◢k◣-ary搜索树的快速求交算法

k-ary搜索树因其对高速缓存和SIMD并行指令集天然的适配性,正在受到越来越多的关注和研究.近年来,它被成功地应用于搜索引擎倒排索引结构中,用于实现高效的查询处理和索引压缩.但基于k-ary搜索树的查询处理算法目前仍处在一种相对简单基础的应用程度,效率提升有限;而且查询算法仅限于元素搜索,大大限制了其适用范围.基于上述观察,研究了基于k-ary搜索树的求交算法,并提出了两种优化技术用于压缩搜索范围以提升查询效率.实验证明,结合不同的遍历方式,优化后的求交算法可以极大地提高查询速度,尤其是针对存储海量数据的长倒排链,配合更大的SIMD寄存器,k-ary搜索树相比于传统求交算法的优势更为明显.     

基于Lucene的分布式并行索引

索引技术是搜索引擎的核心技术之一,索引技术的好坏直接影响到搜索引擎的查准率以及对用户的响应速度.Lucene是一个优秀的全文检索引擎架构,采用高度优化的倒排索引结构并支持增量索引.但在实际应用Lucene时存在一个值得关注的问题:随着被索引文件的增多,索引时间成线性增长,导致建索引的过程会影响搜索体验;在搜索引擎应用中,当索引文件量达到一定等级时,搜索引擎就遇到性能瓶颈.在深入分析和研究Lucene索引机制的基础上,采用以内存为缓冲区建索引文件的分布式并行索引技术形成了一个可扩展的搜索引擎解决方案,极大地缓解了建索引给搜索带来的瓶颈问题.     

基于Lucene的搜索引擎的研究与应用

互联网搜索的精确性一直是衡量搜索引擎性能的重要标志。针对普通搜索引擎的固有缺陷,文中提出了一种应用于新闻检索的搜索引擎。该引擎是利用开源的网络爬虫工具将互联网信息抓取到本地,并利用Lucene开放的API,对特定的信息进行索引和搜索。Lucene是基于Java开发的源代码开放的全文检索工具包,具有高性能、可扩展等特性,是实现搜索引擎的核心组件。通过对Lucene的API进行分析,并在此基础上,构建了索引和搜索的模块,并对网上新闻内容进行实时地搜索。通过与普通搜索引擎对比,该新闻搜索引擎提高了搜索的精确性。     

搜索引擎建立倒排索引的算法研究

搜索引擎并不真正搜索互联网,它搜索的实际上是预先整理好的网页索引数据库.因此网页索引库建立的好坏直接影响最后的查询结果的准确性和用户的查询速度.本文提出了一种建立倒排索引的算法并进行了分析和研究.     

个性化综合倒排索引在Lucene中的应用

该文针对目前通用搜索引擎存在的不足,提出在建立普通倒排索引的基础上,再建立一个记录用户手动标注信息的综合倒排索引,并结合渠道奖励词频算法和文档关注度算法动态更新综合倒排索引,最后在Lucene环境下实现了一个体现用户个性的搜索引擎。     

搜索引擎倒排索引技术的改进

倒排文件是搜索引擎检索系统普遍采用的索引技术。在实验基础上，针对中文搜索引擎中索引的时效性和传统倒排索引在更新时的缺点，提出分组索引技术和一种追加索引的更新算法，可以有效提高搜索引擎的检索效率，同时不影响系统检索效果。     

基于Lucene的中文倒排索引技术的研究

索引是所有搜索引擎的核心概念,为了进行快速查找,就需要把数据处理成一种高效的、可交叉引用的组织格式。倒排索引是一种高效的索引组织模式,其组织模式和存储结构对检索系统的性能起着至关重要的作用,文中探讨了基于Lucene的倒排索引模式,分析了Lucene索引文件的结构、索引过程以及相关排序算法,讨论了Lucene的压缩算法,并且通过设计一个中文模块实现了基于正向减字最大匹配分词方式的中文索引。实验表明新的分词算法比Lucene自带的分词算法性能有了很大的提高。     

倒排链表多层自索引结构的分析与设计

在大型搜索引擎的查询处理过程中,对索引文件的顺序遍历将导致响应时间过长。自索引结构能够有效解决上述问题。为了进一步缩短查询时间,通过对当前索引文件自索引结构的分析,设计了倒排链表的多层自索引结构。此结构以定长元组为单位,使用迭代的方法提取数据段同步点形成上层自索引。在此基础上,实现了索引压缩与查询系统。基于TREC GOV2数据集的测试结果表明,该系统的各项性能指标均优于不采用自索引结构的查询系统,在不影响结果精度的情况下使检索效率得到大幅提升。     

多核平台上Top-k查询的性能优化

随着Web搜索引擎等实际应用的发展,Top-k查询的实时性能日益受人关注.为了提升Top-k查询在高性能计算机上的实时性能,本文针对领域中具有代表性的No Radom Access(NRA)程序进行性能分析,根据多核处理器的结构特点,采用分层的优化方法对NRA程序进行并行优化.通过使用数据结构调整,任务调度等方法优化,优化后程序在实验数据集上获得了59%的串行性能提升,并具有接近线性的加速比.     

搜索引擎的混合索引技术

倒排文件是搜索引擎检索系统普遍采用的索引技术。针对中文搜索引擎中采用自动分词的全文检索因分词词典规模小导致的检索效率下降与词典规模扩大导致检索效果下降的矛盾,论文在天网搜索引擎的实践基础上,提出了一种基于倒排文件实现的混合索引的方法,它可以有效提高搜索引擎下短语查询的检索效率,同时不影响系统检索效果。     

基于FAQ的OTC问答系统的设计与实现

设计并实现了一个基于FAQ(Frequently Asked Question)的OTC(Over The Counter)问答系统。描述了该系统的工作流程、系统结构;详细阐述了系统实现的关键技术,包括基于倒排索引的查找算法、根据用户问题建立候选问题集和基于知网的语义相似度计算方法等。运行结果表明,对于常问问题和普遍性的问题,系统有很高的准确率。     

使用精确索引的P2P搜索

P2P搜索系统需要在节点上分散存放资源的索引,以实现资源的匹配和搜索结果的计算。索引的类型对于如何计算搜索结果和与之相关的性能有很大的影响。目前基于DHT的P2P搜索模型使用的是称之为超集索引的索引类型。该类型的索引不可避免地具有种种性能问题。为此,提出精确索引的概念,以及构建于精确索引之上的P2P搜索模型。该模型达到了相当好的负载平衡性能,并能快速返回搜索结果。     

Inverted index maintenance strategy for flashSSDs: Revitalization of in-place index update strategy

An inverted index is a core data structure of Information Retrieval systems, especially in search engines. Since the search environments have become more dynamic, many on-line index maintenance strategies have been proposed. Previous strategies were designed for HDDs. Consequently, in order to avoid expensive random access cost, Merge-based strategies have been preferred to In-place index update strategies on HDDs. However, flashSSDs have become solid alternatives to HDDs. FlashSSDs currently are adopted in a wide range of areas due to their superior features such as the short access latency, energy efficiency, and high bandwidth. In this article, we first reexamined potentials of In-place index update strategies on flashSSDs. Thanks to the insignificant access latency of flashSSDs, we discovered that In-place index update strategies outperform Merge-based strategies, since In-place index update strategies generate much less amount of I/O than Merge-based strategies despite inducing frequent random accesses. Based on this discovery, we suggest a new inverted index maintenance strategy based on an In-place index update strategy for flashSSDs, called Multipath Flash In-place Strategy (MFIS). To enhance the index maintenance performance, MFIS stores the posting list of each term non-contiguously and exploits the internal parallelism of flashSSDs. Thus, MFIS not only induces the minimum amount of I/O but also utilizes the maximum bandwidth of flashSSDs. Furthermore, MFIS is designed to show high query processing performance by utilizing the internal parallelism of flashSSDs even though the posting list of each term is stored non-contiguously. In our experiments, the index maintenance performance of MFIS was considerably better than other previous maintenance strategies. The index maintenance performance was up to 14.93, 4.04, 5.12, and 2.33 times higher than Merge-based strategies such as Immediate Merge, Geometric Partitioning, Hybrid, and SSD-aware Hybrid, respectively. The query processing performance of MFIS was up to 1.62 times higher than non-contiguous In-place. In addition, MFIS showed almost the best query processing performance as Merge-based strategies did. In conclusion, MFIS is the best on-line inverted index maintenance strategy on flashSSDs in terms of both index maintenance and query processing performance.     

Three-Level Caching for Efficient Query Processing in Large Web Search Engines

Large web search engines have to answer thousands of queries per second with interactive response times. Due to the sizes of the data sets involved, often in the range of multiple terabytes, a single query may require the processing of hundreds of megabytes or more of index data. To keep up with this immense workload, large search engines employ clusters of hundreds or thousands of machines, and a number of techniques such as caching, index compression, and index and query pruning are used to improve scalability. In particular, two-level caching techniques cache results of repeated identical queries at the frontend, while index data for frequently used query terms are cached in each node at a lower level. We propose and evaluate a three-level caching scheme that adds an intermediate level of caching for additional performance gains. This intermediate level attempts to exploit frequently occurring pairs of terms by caching intersections or projections of the corresponding inverted lists. We propose and study several offline and online algorithms for the resulting weighted caching problem, which turns out to be surprisingly rich in structure. Our experimental evaluation based on a large web crawl and real search engine query log shows significant performance gains for the best schemes, both in isolation and in combination with the other caching levels. We also observe that a careful selection of cache admission and eviction policies is crucial for best overall performance. Work supported by NSF CAREER Award CCR-0093400 and the New York State Center for Advanced Technology in Telecommunications (CATT) at Polytechnic University.     

SKIF-P: a point-based indexing and ranking of web documents for spatial-keyword search

There is a significant commercial and research interest in location-based web search engines. Given a number of search keywords and one or more locations (geographical points) that a user is interested in, a location-based web search retrieves and ranks the most textually and spatially relevant web pages. In this type of search, both the spatial and textual information should be indexed. Currently, no efficient index structure exists that can handle both the spatial and textual aspects of data simultaneously and accurately. Existing approaches either index space and text separately or use inefficient hybrid index structures with poor performance and inaccurate results. Moreover, most of these approaches cannot accurately rank web-pages based on a combination of space and text and are not easy to integrate into existing search engines. In this paper, we propose a new index structure called Spatial-Keyword Inverted File for Points to handle point-based indexing of web documents in an integrated/efficient manner. To seamlessly find and rank relevant documents, we develop a new distance measure called spatial tf-idf. We propose four variants of spatial-keyword relevance scores and two algorithms to perform top-k searches. As verified by experiments, our proposed techniques outperform existing index structures in terms of search performance and accuracy.