一种使用双阈值的数据仓库环境下重复记录消除算法

重复记录消除是数据清理研究中一个很重要的方面,它的目的是检测并消除那些冗余的、可能对后来的OLAP和数据挖掘造成影响的数据。已有研究都是通过设定一个相似度阈值来判断两条记录是否为重复记录。过大的阈值将导致返回率下降,过小的阈值将导致误检率上升。文章提出了一种双阈值的重复记录消除方法,利用数据仓库环境下数据库表之间的外键联系做进一步判断,可以有效地提高判断质量,减小误检率。

Duplicate Records Elimination in Data Warehouse with Two Thresholds

Data Warehouse integrates the data from several data sources.But the data integrated from outer data sources may contain error information such as typical errors or duplicate records.So these data should be cleaned before OLAP or data mining.Duplicate records elimination is an important issue in data cleaning.It is used to detect those duplicate records,which are not identical but may represent the same object.A new algorithm using two thresholds is introduced to detect duplicate records in data warehouse.In this algorithm the relationships such as foreign key reference between two tables are used to help to decide whether two records are duplicates when some traditional algorithms could not make an explicit decision.