基于DOM的Web信息提取

当前，Web已经成为人们获取信息的主要渠道之一。然而，用于表达Web页面信息的HTML语言存在着与生俱来的缺点。HTML的“标记”只是告诉浏览器软件如何显示所定义的信息，却不包含任何语义。因此由HTML语言所表述的Web页面经过浏览器分析后只适合人们浏览，不适合作为一种数据交换的方式由机器处理。该文以文档对象模型DOM为基础，把所要提取的信息在DOM层次结构中的路径作为信息抽取的“坐标”，并以这个基本原理为基础设计了一种归纳学习算法来半自动地生成提取规则，然后根据提取规则生成Java类.生成的Java类可以作为Web数据源包装器组成的重要构件。     

构造Web文档中半结构化信息的技术

为了对Ｗｅｂ上不规则的、动态的信息按照数据库的方式集成和查询,采用对象交换模型（ＯＥＭ）建立了Ｗｅｂ上信息模型。为了将页面中各个部分表示为应用的ＯＥＭ对象,设计了半结构化信息的抽取算法,并给出测试结果。该方法可以抽取结构化和半结构化的信息,比现有的抽取方法通用性更强。     

Semantic Web search based on ontological conjunctive queries

Many experts predict that the next huge step forward in Web information technology will be achieved by adding semantics to Web data, and will possibly consist of (some form of) the Semantic Web. In this paper, we present a novel approach to Semantic Web search, called Serene, which allows for a semantic processing of Web search queries, and for evaluating complex Web search queries that involve reasoning over the Web. More specifically, we first add ontological structure and semantics to Web pages, which then allows for both attaching a meaning to Web search queries and Web pages, and for formulating and processing ontology-based complex Web search queries (i.e., conjunctive queries) that involve reasoning over the Web. Here, we assume the existence of an underlying ontology (in a lightweight ontology language) relative to which Web pages are annotated and Web search queries are formulated. Depending on whether we use a general or a specialized ontology, we thus obtain a general or a vertical Semantic Web search interface, respectively. That is, we are actually mapping the Web into an ontological knowledge base, which then allows for Semantic Web search relative to the underlying ontology. The latter is then realized by reduction to standard Web search on standard Web pages and logically completed ontological annotations. That is, standard Web search engines are used as the main inference motor for ontology-based Semantic Web search. We develop the formal model behind this approach and also provide an implementation in desktop search. Furthermore, we report on extensive experiments, including an implemented Semantic Web search on the Internet Movie Database.     

Databases deepen the Web

The Web has become the preferred medium for many database applications, such as e-commerce and digital libraries. These applications store information in huge databases that users access, query, and update through the Web. Database-driven Web sites have their own interfaces and access forms for creating HTML pages on the fly. Web database technologies define the way that these forms can connect to and retrieve data from database servers. The number of database-driven Web sites is increasing exponentially, and each site is creating pages dynamically-pages that are hard for traditional search engines to reach. Such search engines crawl and index static HTML pages; they do not send queries to Web databases. The information hidden inside Web databases is called the "deep Web" in contrast to the "surface Web" that traditional search engines access easily. We expect deep Web search engines and technologies to improve rapidly and to dramatically affect how the Web is used by providing easy access to many more information resources.     

Embedded grammar tags: advancing natural language interaction on the Web

Embedded grammar tags (EGTs) reflect Web page content by anticipating queries users might launch to retrieve that content. The grammars provide a unifying component for speech recognition engines, semantic Web page representation, and speech output generation. We propose a new framework that lets intelligent agents discover accurate, concise responses to natural language queries. This framework's backbone consists of embedded grammar tags that capture natural language queries. EGTs reflect information content in Web pages by anticipating the queries that users might launch to retrieve particular content.     

Enriching the conceptual basis for query formulation through relationship semantics in databases

The rapid increase in end-user computing calls into question the suitability of existing database query languages (DBQLs). Because the typical DB end-user is not a DB specialist, it is essential that DBQLs use concepts that are as close as possible to those in the end-users’ cognitive mental model and adopt interface techniques that are suited to end-users’ abilities. Concept-based query languages are well suited for this. This realization has motivated further research in conceptual, or semantic, query approaches. However, the primary focus in this field has been on semantic query optimization, not on query formulation. In this study, we address ourselves to the problem of formulation of queries using concepts. We propose a concept-based query language, called the conceptual query language (CQL), which allows for the conceptual abstraction of database queries and exploits the rich semantics of data models to ease and facilitate query formulation.The CQL approach uses the relationship semantics of semantic data models to render transparent the technical complexities of existing DB query languages. Association semantics are also used to automatically construct query graphs and pseudo-natural language explanations of queries, and to generate SQL codes. A set theoretic formalism for conceptual queries is developed and used. This paper discusses the design of CQL, its expressive power, its implementation, and the strategies for CQL query processing. The implementation of a CQL prototype is briefly discussed in this paper. User experiments were carried out extensively and showed the advantage of CQL over alternative languages such as SQL.     

Learning Rules for Conceptual Structure on the Web

This paper presents an infrastructure and methodology to extract conceptual structure from Web pages, which are mainly constructed by HTML tags and incomplete text. Human beings can easily read Web pages and grasp an idea about the conceptual structure of underlying data, but cannot handle excessive amounts of data due to lack of patience and time. However, it is extremely difficult for machines to accurately determine the content of Web pages due to lack of understanding of context and semantics. Our work provides a methodology and infrastructure to process Web data and extract the underlying conceptual structure, in particular relationships between ontological concepts using Inductive Logic Programming in order to help with automating the processing of the excessive amount of Web data by capturing its conceptual structures.     

数据库语义合取查询研究

针对异质、异构数据库的语义集成中,对海量元组进行语义查询时因效率问题而无法使用丰富的语义表达能力的问题,提出一种兼顾速度和语义表达能力的算法,将语义查询和本体都进行图形化表示,并实现子图的语义匹配,将匹配的结果转化成数据库查询语句。与将语义查询直接重写为SQL的算法相比,该算法能支持更丰富的语义。     

基于网页上下文的Deep Web数据库分类

讨论了提高Deep Web数据库分类准确性的若干新技术,其中包括利用HTML网页的内容文本作为理解数据库内容的上下文和把数据库表的属性标记词归一的过程.其中对网页中的内容文本的发现算法是基于对网页文本块的多种统计特征.而对数据库属性标记词的归一过程是把同义标记词用代表词进行替代的过程.给出了采用分层模糊集合对给定学习实例所发现的领域和语言知识进行表示和基于这些知识对标记词归一化算法.基于上述预处理,给出了计算Deep Web数据库的K-NN(k nearest neighbors)分类算法,其中对数据库之间语义距离计算综合了数据库表之间和含有数据库表的网页的内容文本之间的语义距离.分类实验给出算法对未预处理的网页和经过预处理后的网页在数据库分类精度、查全率和综合F1等测度上的分类结果比较.     

Beyond search: Retrieving complete tuples from a text-database

A common task of Web users is querying structured information from Web pages. For realizing this interesting scenario we propose a novel query processor for systematically discovering instances of semantic relations in Web search results and joining these relation instances into complex result tuples with conjunctive queries. Our query processor transforms a structured user query into keyword queries that are submitted to a search engine, forwards search results to a relation extractor, and then combines relations into complex result tuples. The processor automatically learns discriminative and effective keywords for different types of semantic relations. Thereby, our query processor leverages the index of a search engine to query potentially billions of pages. Unfortunately, relation extractors may fail to return a relation for a result tuple. Moreover, user defined data sources may not return at least k complete result tuples. Therefore we propose an adaptive routing model based on information theory for retrieving missing attributes of incomplete result tuples. The model determines the most promising next incomplete tuple and attribute type for returning any-k complete result tuples at any point during the query execution process. We report a thorough experimental evaluation over multiple relation extractors. Our query processor returns complete result tuples while processing only very few Web pages.     

Improving large-scale search engines with semantic annotations

Traditional search engines have become the most useful tools to search the World Wide Web. Even though they are good for certain search tasks, they may be less effective for others, such as satisfying ambiguous or synonym queries. In this paper, we propose an algorithm that, with the help of Wikipedia and collaborative semantic annotations, improves the quality of web search engines in the ranking of returned results. Our work is supported by (1) the logs generated after query searching, (2) semantic annotations of queries and (3) semantic annotations of web pages. The algorithm makes use of this information to elaborate an appropriate ranking. To validate our approach we have implemented a system that can apply the algorithm to a particular search engine. Evaluation results show that the number of relevant web resources obtained after executing a query with the algorithm is higher than the one obtained without it.     

Turning tables [HTML tables]

Behind all the fancy tools that churn out volumes of messy HTML, there is a basic set of standardized HTML tags that all professional Webspinners must learn to master. It is important to understand these tags and how they work because despite recent advances in code-checking tools, debugging HTML often still comes down to hand-tweaking code with simple text editors like NotePad and BBEdit. The paper discusses HTML tables which are the fundamental building blocks for most of today's Web pages. It considers three basic tags to build a table     

用于网页相关性判断的聚焦查询文摘算法研究

为了进一步提高网页相关性判断的速度和准确率,提出了一种新的用于聚焦文摘的句子权重计算方法。在查询返回的结果集的基础上,通过计算关键词间的互信息,对输入的查询语句进行短语识别;利用网页文本中的标签信息,对网页结构进行分析,并将关键词短语和网页结构等信息融入句子权重计算。实验结果表明,基于该算法生成的查询摘要在相关性判断的速度和准确率等方面均优于现有方法。     

Web Data Extraction from Query Result Pages Based on Visual and Content Features

A rapidly increasing number of Web databases are now become accessible via their HTML form-based query interfaces. Query result pages are dynamically generated in response to user queries, which encode structured data and are displayed for human use. Query result pages usually contain other types of information in addition to query results, e.g., advertisements, navigation bar etc. The problem of extracting structured data from query result pages is critical for web data integration applications, such as comparison shopping, meta-search engines etc, and has been intensively studied. A number of approaches have been proposed. As the structures of Web pages become more and more complex, the existing approaches start to fail, and most of them do not remove irrelevant contents which may affect the accuracy of data record extraction. We propose an automated approach for Web data extraction. First, it makes use of visual features and query terms to identify data sections and extracts data records in these sections. We also represent several content and visual features of visual blocks in a data section, and use them to filter out noisy blocks. Second, it measures similarity between data items in different data records based on their visual and content features, and aligns them into different groups so that the data in the same group have the same semantics. The results of our experiments with a large set of Web query result pages in di?erent domains show that our proposed approaches are highly effective.     

Web文档清洗系统中HTML解析器的开发

对于组建一个面向Web的信息系统来说 ,去除掉脚本、广告链接以及导航链接等无用数据 ,将提高信息存储和检索的效率 ;同时 ,基于语义对Web文档进行合并和分割也会有助于信息的管理 ,这些都是Web文档清洗系统的任务。在Web文档清洗中 ,无论是脱机的规则学习还是联机的文档清洗 ,都需要建立在对Web文档的结构和内容进行分析的基础之上。从HTML解析的一般概念入手 ,结合Web文档清洗系统的需求 ,描述了一个自主开发的HTML解析器的结构 ,并对其组成部分 :词典、词法分析器和语法分析器的设计作了详细的讨论     

Dynamic Query Optimization Approach for Semantic Database Grid

Fundamentally, semantic grid database is about bringing globally distributed databases together in order to coordinate resource sharing and problem solving in which information is given well-defined meaning, and DartGrid II is the implemented database gird system whose goal is to provide a semantic solution for integrating database resources on the Web. Although many algorithms have been proposed for optimizing query-processing in order to minimize costs and/or response time, associated with obtaining the answer to query in a distributed database system, database grid query optimization problem is fundamentally different from traditional distributed query optimization. These differences are shown to be the consequences of autonomy and heterogeneity of database nodes in database grid. Therefore, more challenges have arisen for query optimization in database grid than traditional distributed database. Following this observation, the design of a query optimizer in DartGrid II is presented, and a heuristic, dynamic and parallel query optimization approach to processing query in database grid is proposed. A set of semantic tools supporting relational database integration and semantic-based information browsing has also been implemented to realize the above vision.     

基于结构一致和特征学习的网页信息标签提取

网页信息指网页的正文、标题、发布时间、媒体等,每个信息都存在于HTML文档特定的标签中,自动获取这些标签可以实现在相同模板下的网页信息自动提取,对于大规模抓取网页内容有很大帮助。由于在相同模板下不同网页之间结构一致,网页信息有一定统计特征,提出了一种基于结构对比和特征学习的网页信息标签自动提取算法。该算法包含三个步骤：网页对比、内容识别和标签提取。在51个模块下对1?620个网页进行测试,实验结果表明,通过提取标签获取网页信息不仅速度快,而且抓取的内容更加准确。     

Programming with logical queries, bulk updates, and hypotheticalreasoning

The paper presents a language of update programs that integrates logical queries, bulk updates and hypothetical reasoning in a seamless manner. There is no syntactic or semantic distinction between queries and updates. Update programs extend logic programs with negation in both syntax and semantics. They allow bulk updates in which an arbitrary update is applied simultaneously for all answers of an arbitrary query. Hypothetical reasoning is naturally supported by testing the success or failure of an update. We describe an alternating fixpoint semantics of update programs and show that it can express all nondeterministic database transformations     

基于COM的通用数据库访问组件的设计与实现

给出了一个基于COM的通用数据库访问组件的设计和实现方法，提出了一种动态生成HTML网页的算法。组件通过接收浏览器提交的参数进行数据库操作，把结果以HTML页面的方式反馈给浏览器，从而实现浏览器和数据库之间的交互式操作。