Bloom Filters散列函数数目多阶段动态优化算法

标准Bloom Filters在操作前需要知道数据集合中不同元素数目才能确定最佳的Hash函数数目,但是数据集的分布情况并不容易事先获得.本文提出一种多阶段Hash函数数目动态优化的Bloom Filters(Multi-stage Dynamic optimization Bloom Filters,MDBF),它将...     

基于布隆过滤器的字符串模糊匹配算法的FPGA实现

深度包检测技术通过对数据包内容的深入扫描和检测,能够有效识别出隐藏在数据包有效载荷内的非法数据,但该技术存在功耗非常大的缺点。针对该问题,提出了采用Bloom Filter(布隆过滤器)进行字符串模糊匹配方式,利用Bloom Filter将信息流中大部分正常流量过滤掉,从而减轻了后端的字符串精确匹配的压力,降低了系统功耗,大大提高了处理速度。     

关于布隆过滤器在BSS中应用

介绍布隆过滤器(Bloom Filter)的相关算法原理和使用说明,并阐述其在BSS领域中应用。通过与Redis缓存技术相结合,利用布隆过滤器(Boom Filter)的高效匹配、低存储等优势,提高BSS中排重效率,减少BSS对硬件扩容的需求。同时,阐述BSS排重中关于位数组的划分,以及针对布隆过滤器(Bloom Filter)对数据存在一定误判率的不足,并提出相应的应对措施。     

Performance evaluation and comparison of three counting bloom filter schemes

The Counting Bloom Filter (CBF) is a kind of space-efficient data structure that extends a Bloom filter so as to allow approximate multiplicity queries on a dynamic multi-set. This paper evaluates the performance of multiplicity queries of three simple CBF schemes–the Naïve Counting Bloom Filter (NCBF), the Space-Code Bloom Filter (SCBF) and the d-left Counting Bloom Filter (dlCBF)–using metrics of space complexity and counting error under both uniform and zipfian mul-tiplicity distributions. We compare their counting error under same space complexity, and their space complexity when similar counting errors are achieved respectively. Our results show that dlCBF is the best while SCBF is the worst in terms of both space-efficiency and accuracy. Furthermore, the per-formance gap between dlCBF and the others has a trend of being enlarged with the increment of space occupation or counting accuracy.     

一种基于并行Bloom Filter的高速URL查找算法

URL查找是众多网络系统中重要的组成部分,如URL过滤系统、Web缓存等.随着互联网的迅速发展,URL查找面临的主要挑战是实现大规模URL集合下的高速查找,同时保证低存储和低功耗.本文提出了一种基于并行Bloom Filter的URL查找算法,CaBF.该算法高度并行化,提供大规模URL集合下的高速最长前缀匹配,并很好地适应集合中不同数量的URL组件.理论分析和真实网络数据集上的实验表明,该算法相比现有算法可以降低假阳性概率达一个数量级(或者在满足相同假阳性概率的前提下降低存储和硬件逻辑资源消耗).此外,该方法的体系结构很容易映射到FPGA等硬件器件上,提供每秒超过150M次的URL查找速度.     

基于自适应超时计数布鲁姆过滤器的流量测量算法

针对流量测量中IP长流的检测问题,该文设计了计数布鲁姆过滤器(Count Bloom Filter, CBF)与超时布鲁姆过滤器(Timeout Bloom Filter, TBF)结合的长流检测机制。该机制动态调整布鲁姆过滤器中的超时时间,及时清理结束流,解决空间拥塞问题,从而可以适用于无结束标志IP长流检测。依据算法整体错误率与超时时间的分析,根据链路流到达强度与布鲁姆过滤器向量空间长度自适应动态调整超时时间,使得算法整体错误率保持最低。该算法的性能利用真实网络流量数据进行验证,结果表明,与现有算法相比,该算法的测量准确性更高。     

一种距离加权Bloom Filter的网络路由机制

基于分布式哈希表（DHT）的P2P查找经常受到在底层网络中路由时无必要的路径长度增加的影响。另外．DHT在处理复制方面也有一定的缺陷。提出了距离加权Bloom Filter（dwBF），详细地阐述了在资源分散的覆盖网络中使用距离加权Bloom Filter网络路由算法。     

Bloom Filter研究进展

近年来,由于Bloom filter具有可压缩性和高效查询性,其在分布式数据库、网络缓存、对等网和信息检索等领域引起了越来越多的研究者关注。随着Bloom filter不同应用需求的出现,多种Bloom filter变体被提了出来,诸如:支持删除元素的CBF;可以统计频次型的SBF、DCF、dlCBF;大小可以动态伸长的DBF、SBF;压缩型BF等。本文对Bloom filter及其各种变体进行了介绍,并对其特点进行了分析比较,总结了它们各自的优势和不足,并进一步指出了Bloom filter未来的一些研究方向。     

一种无线传感器网络的访问控制协议

传感器网络中的节点存在由于能量耗尽或恶意攻击而丧失作用的威胁,因此需要新节点的加入.利用Bloom Filter技术,提出了一种访问控制协议.本协议不仅便于实现新节点、旧节点的双向认证和密钥协商,而且便于实现节点的加入与撤消.通过性能分析和安全性分析说明了该协议的有效性.     

基于计数布鲁姆过滤器的集合调和算法

提出了一种基于计数布鲁姆过滤器的集合调和方法,该方法将远程节点A和B上的数据集合SA和SB分别用计数布鲁姆过滤器表示,设为CBF(SA)和CBF(SB),节点A将CBF(SA)发送给节点B,节点B进行计数布鲁姆过滤器减运算,得到差过滤器CBF(SB)CBF(SA),然后利用CBF(SB)CBF(SA)查询并获得集合SB中的差集元素SB SA,并返回SB SA给节点A,最后节点A用差集SB SA和自身集合SA进行集合并运算,完成集合调和。由于计数布鲁姆过滤器支持集合元素的删除操作,因此,该方法非常适合应用于数据集合更新频繁的分布式系统。理论分析和仿真实验结果表明,该方法既具有精确集合调和,能得到全部SB SA元素的优点,也具有近似集合调和仅需单轮消息交换的优点。     

Compressed Bloom filters

A Bloom filter is a simple space-efficient randomized data structure for representing a set in order to support membership queries. Although Bloom filters allow false positives, for many applications the space savings outweigh this drawback when the probability of an error is sufficiently low. We introduce compressed Bloom filters, which improve performance when the Bloom filter is passed as a message, and its transmission size is a limiting factor. For example, Bloom filters have been suggested as a means for sharing Web cache information. In this setting, proxies do not share the exact contents of their caches, but instead periodically broadcast Bloom filters representing their caches. By using compressed Bloom filters, proxies can reduce the number of bits broadcast, the false positive probability, and/or the amount of computation per lookup. The cost is the processing time for compression and decompression, which can use simple arithmetic coding, and more memory use at the proxies, which utilize the larger uncompressed form of the Bloom filter.     

面向大数据处理的高精度多维计数布鲁姆过滤器

分析了现有多维布鲁姆过滤器查询算法的工作原理和特点,针对大数据处理特点提出了一种基于双射函数的高精度多维计数布鲁姆过滤器(AMD-CBF)查询算法.AMD-CBF中元素表示和查找分两步进行,第1步将元素各属性哈希映射到各自对应的高精度计数布鲁姆过滤器(A-CBF)中;第2步将元素的所有属性通过双射函数转换为一个值来表示元素整体信息,然后将这个值哈希映射到联合计数布鲁姆过滤器中(C-CBF),完成元素整体的表示和查询确认.理论分析和仿真实验结果表明,AMD-CBF能够支持多维集合元素的高效表示和查询及删除,相比同类研究查询假阳性降低明显,查询精度大幅度提高.     

基于二叉树SVM的弹道目标RCS识别

二叉树支持向量机(SVM)是一种针对多类问题的有效分类器,具有结构简单、训练快的特点,但二叉树SVM容易出现误差积累,且不能输出识别结果的置信度。文中设计了一种基于隶属度计算的二叉树SVM分类器,首先,该分类器利用方差和最小准则选择节点,将多类问题转化为偏二叉树SVM分类问题,避免了误差积累,然后,利用特征变换空间的类中心和类半径,计算出样本结果的置信度,使得二叉树SVM分类器能够输出模糊结果。将上述二叉树SVM分类器应用于弹道目标的RCS特征识别,仿真结果表明了该方法的有效性。     

Longest prefix matching using bloom filters

We introduce the first algorithm that we are aware of to employ Bloom filters for longest prefix matching (LPM). The algorithm performs parallel queries on Bloom filters, an efficient data structure for membership queries, in order to determine address prefix membership in sets of prefixes sorted by prefix length. We show that use of this algorithm for Internet Protocol (IP) routing lookups results in a search engine providing better performance and scalability than TCAM-based approaches. The key feature of our technique is that the performance, as determined by the number of dependent memory accesses per lookup, can be held constant for longer address lengths or additional unique address prefix lengths in the forwarding table given that memory resources scale linearly with the number of prefixes in the forwarding table. Our approach is equally attractive for Internet Protocol Version 6 (IPv6) which uses 128-bit destination addresses, four times longer than IPv4. We present a basic version of our approach along with optimizations leveraging previous advances in LPM algorithms. We also report results of performance simulations of our system using snapshots of IPv4 BGP tables and extend the results to IPv6. Using less than 2 Mb of embedded RAM and a commodity SRAM device, our technique achieves average performance of one hash probe per lookup and a worst case of two hash probes and one array access per lookup.     

Space-Code Bloom Filter for Efficient Per-Flow Traffic Measurement

Per-flow traffic measurement is critical for usage accounting, traffic engineering, and anomaly detection. Previous methodologies are either based on random sampling (e.g., Cisco's NetFlow), which is inaccurate, or only account for the "elephants." We introduce a novel technique for measuring per-flow traffic approximately, for all flows regardless of their sizes, at very high-speed (say, OC768). The core of this technique is a novel data structure called Space-Code Bloom Filter (SCBF). A SCBF is an approximate representation of a multiset; each element in this multiset is a traffic flow and its multiplicity is the number of packets in the flow. The multiplicity of an element in the multiset represented by SCBF can be estimated through either of two mechanisms-maximum-likelihood estimation or mean value estimation. Through parameter tuning, SCBF allows for graceful tradeoff between measurement accuracy and computational and storage complexity. SCBF also contributes to the foundation of data streaming by introducing a new paradigm called blind streaming. We evaluate the performance of SCBF through mathematical analysis and through experiments on packet traces gathered from a tier-1 ISP backbone. Our results demonstrate that SCBF achieves reasonable measurement accuracy with very low storage and computational complexity. We also demonstrate the application of SCBF in estimating the frequency of keywords at a search engine-demonstrating the applicability of SCBF to other problems that can be reduced to multiset membership queries     

两层传感器网络中一种高效的加密数据条件聚合协议研究

提出了一种隐私保护的条件聚合协议,使存储节点在不知道数据真实值的情况下对满足条件的数据进行聚合,防止存储节点对敏感信息的泄漏。为了保护数据和查询条件的隐私性,提出了一种基于前缀成员确认和布鲁姆过滤器相结合的编码方法对数据和查询条件进行编码,实现存储节点在不知道数据真实值和查询条件真实值的情况下进行查询处理;为了对查询结果中的数据进行聚合而不暴露数据真实值,采用同态加密技术对数据进行加密,使数据在不解密的情况下能进行聚合运算。进一步,根据传感器采集数据的特点,提出了一种基于代码表的数据压缩表示及传输方法,有效减小了传感器节点和存储节点之间的通信开销。分析和实验结果验证了所提方案的有效性。     

布鲁姆过滤器代数运算探讨

 本文探讨布鲁姆过滤器的代数运算和集合查询的关系,定义布鲁姆过滤器的"并","交","异或","补","差"代数运算,从理论和实验两方面分析布鲁姆过滤器的代数运算和集合代数运算并集,交集,异或集,补集,差集的元素查询关系.理论分析和实验结果表明,布鲁姆过滤器的"并","交"运算能够支持集合并集交集的元素查询,这一结论可以简化利用布鲁姆过滤器进行的系统设计.     

基于流重复信息检测来估寻服务节点

Discovery of service nodes in flows is a challenging task, especially in large ISPs or campus networks where the amount of traffic across network is massive. We propose an effective data structure called Round-robin Buddy Bloom Filters (RBBF) to detect duplicate elements in flows. A two-stage approximate algorithm based on RBBF which can be used for detecting service nodes from NetFlow data is also given and the performance of the algorithm is analyzed. In our case, the proposed algorithm uses about 1% memory of hash table with false positive error rate less than 5% . A prototype system, which is compatible with both IPv4 and IPv6, using the proposed data structure and algorithm is introduced. Some real world case studies based on the prototype system are discussed.