首页 | 官方网站   微博 | 高级检索  
     


Towards adaptive learning with improved convergence of deep belief networks on graphics processing units
Authors:Noel Lopes  Bernardete Ribeiro
Affiliation:1. CISUC – Center for Informatics and Systems of University of Coimbra, Portugal;2. UDI/IPG – Research Unit, Polytechnic of Guarda, Portugal;3. Department of Informatics Engineering, University of Coimbra, Portugal
Abstract:In this paper we focus on two complementary approaches to significantly decrease pre-training time of a deep belief network (DBN). First, we propose an adaptive step size technique to enhance the convergence of the contrastive divergence (CD) algorithm, thereby reducing the number of epochs to train the restricted Boltzmann machine (RBM) that supports the DBN infrastructure. Second, we present a highly scalable graphics processing unit (GPU) parallel implementation of the CD-k algorithm, which boosts notably the training speed. Additionally, extensive experiments are conducted on the MNIST and the HHreco databases. The results suggest that the maximum useful depth of a DBN is related to the number and quality of the training samples. Moreover, it was found that the lower-level layer plays a fundamental role for building successful DBN models. Furthermore, the results contradict the pre-conceived idea that all the layers should be pre-trained. Finally, it is shown that by incorporating multiple back-propagation (MBP) layers, the DBNs generalization capability is remarkably improved.
Keywords:Deep learning  Deep belief networks  Restricted Boltzmann machines  Contrastive divergence  Adaptive step size  GPU computing
本文献已被 ScienceDirect 等数据库收录!
设为首页 | 免责声明 | 关于勤云 | 加入收藏

Copyright©北京勤云科技发展有限公司    京ICP备09084417号-23

京公网安备 11010802026262号