Boundedness and Convergence of Split-Complex Back-Propagation Algorithm with Momentum and Penalty

This paper investigates the split-complex back-propagation algorithm with momentum and penalty for training complex-valued neural networks. Here the momentum are used to accelerate the convergence of the algorithm and the penalty are used to control the magnitude of the network weights. The sufficient conditions for the learning rate, the momentum factor, the penalty coefficient, and the activation functions are proposed to establish the theoretical results of the algorithm. We theoretically prove the boundedness of the network weights during the training process, which is usually used as a precondition for convergence analysis in literatures. The monotonicity of the error function and the convergence of the algorithm are also guaranteed.     

Boundedness and Convergence of Online Gradient Method With Penalty for Feedforward Neural Networks

In this brief, we consider an online gradient method with penalty for training feedforward neural networks. Specifically, the penalty is a term proportional to the norm of the weights. Its roles in the method are to control the magnitude of the weights and to improve the generalization performance of the network. By proving that the weights are automatically bounded in the network training with penalty, we simplify the conditions that are required for convergence of online gradient method in literature. A numerical example is given to support the theoretical analysis.      

Boundedness and Convergence of Online Gradient Method with Penalty for Linear Output Feedforward Neural Networks

This paper investigates an online gradient method with penalty for training feedforward neural networks with linear output. A usual penalty is considered, which is a term proportional to the norm of the weights. The main contribution of this paper is to theoretically prove the boundedness of the weights in the network training process. This boundedness is then used to prove an almost sure convergence of the algorithm to the zero set of the gradient of the error function.     

Convergence of an online gradient method with inner-product penalty and adaptive momentum

In this paper, we study the convergence of an online gradient method with inner-product penalty and adaptive momentum for feedforward neural networks, assuming that the training samples are permuted stochastically in each cycle of iteration. Both two-layer and three-layer neural network models are considered, and two convergence theorems are established. Sufficient conditions are proposed to prove weak and strong convergence results. The algorithm is applied to the classical two-spiral problem and identification of Gabor function problem to support these theoretical findings.     

递归pi-sigma神经网络的带惩罚项的梯度算法分析

传统的梯度算法存在收敛速度过慢的问题,针对这个问题,提出一种将惩罚项加到传统误差函数的梯度算法以训练递归pi-sigma神经网络,算法不仅提高了神经网络的泛化能力,而且克服了因网络初始权值选取过小而导致的收敛速度过慢的问题,相比不带惩罚项的梯度算法提高了收敛速度。从理论上分析了带惩罚项的梯度算法的收敛性,并通过实验验证了算法的有效性。     

Pi-sigma神经网络的乘子法随机单点在线梯度算法*

在利用梯度算法训练Pi-sigma神经网络时,存在因权值选取过小导致收敛速度过慢的问题,而采用一般罚函数法虽然可以克服这个缺点,但要求罚因子必须趋近于∞且惩罚项绝对值不可微,从而导致数值求解困难。为克服以上缺点,提出了一种基于乘子法的随机单点在线梯度算法。利用最优化理论方法,将有约束问题转换为无约束问题,利用乘子法来求解网络误差函数。从理论上分析了算法的收敛速度和稳定性,仿真实验结果验证了算法的有效性。     

基于条件梯度Wasserstein生成对抗网络的图像识别

生成式对抗网络GAN功能强大,但是具有收敛速度慢、训练不稳定、生成样本多样性不足等缺点。该文结合条件深度卷积对抗网络CDCGAN和带有梯度惩罚的Wasserstein生成对抗网络WGAN-GP的优点,提出了一个混合模型-条件梯度Wasserstein生成对抗网络CDCWGAN-GP,用带有梯度惩罚的Wasserstein距离训练对抗网络保证了训练稳定性且收敛速度更快,同时加入条件c来指导数据生成。另外为了增强判别器提取特征的能力,该文设计了全局判别器和局部判别器一起打分,最后提取判别器进行图像识别。实验结果证明,该方法有效的提高了图像识别的准确率。     

Training pi-sigma network by online gradient algorithm with penalty for small weight update

A pi-sigma network is a class of feedforward neural networks with product units in the output layer. An online gradient algorithm is the simplest and most often used training method for feedforward neural networks. But there arises a problem when the online gradient algorithm is used for pi-sigma networks in that the update increment of the weights may become very small, especially early in training, resulting in a very slow convergence. To overcome this difficulty, we introduce an adaptive penalty term into the error function, so as to increase the magnitude of the update increment of the weights when it is too small. This strategy brings about faster convergence as shown by the numerical experiments carried out in this letter.     

A Hybrid Projection-based and Radial Basis Function Architecture: Initial Values and Global Optimisation

We introduce a mechanism for constructing and training a hybrid architecture of projection-based units and radial basis functions. In particular, we introduce an optimisation scheme which includes several steps and assures a convergence to a useful solution. During network architecture construction and training, it is determined whether a unit should be removed or replaced. The resulting architecture often has a smaller number of units compared with competing architectures. A specific overfitting resulting from shrinkage of the RBF radii is addressed by introducing a penalty on small radii. Classification and regression results are demonstrated on various benchmark data sets and compared with several variants of RBF networks [1,2]. A striking performance improvement is achieved on the vowel data set [3]. Received: 03 November 2000, Received in revised form: 25 October 2001, Accepted: 04 January 2002     

Stochastic optimisation with inequality constraints using simultaneous perturbations and penalty functions

We present a stochastic approximation algorithm based on penalty function method and a simultaneous perturbation gradient estimate for solving stochastic optimisation problems with general inequality constraints. We present a general convergence result that applies to a class of penalty functions including the quadratic penalty function, the augmented Lagrangian, and the absolute penalty function. We also establish an asymptotic normality result for the algorithm with smooth penalty functions under minor assumptions. Numerical results are given to compare the performance of the proposed algorithm with different penalty functions.     

A penalty method for history-dependent variational–hemivariational inequalities

Penalty methods approximate a constrained variational or hemivariational inequality problem through a sequence of unconstrained ones as the penalty parameter approaches zero. The methods are useful in the numerical solution of constrained problems, and they are also useful as a tool in proving solution existence of constrained problems. This paper is devoted to a theoretical analysis of penalty methods for a general class of variational–hemivariational inequalities with history-dependent operators. Unique solvability of penalized problems is shown, as well as the convergence of their solutions to the solution of the original history-dependent variational–hemivariational inequality as the penalty parameter tends to zero. The convergence result proved here generalizes several existing convergence results of penalty methods. Finally, the theoretical results are applied to examples of history-dependent variational–hemivariational inequalities in mathematical models describing the quasistatic contact between a viscoelastic rod and a reactive foundation.     

基于非凸的全变分和低秩混合正则化的图像去模糊模型和算法

非盲图像去模糊问题是从已知核的带噪声的线性卷积变换中恢复原始图像.如果噪声是满足高斯分布的,则可以直接使用最小二乘求解.然而在大多数情况下,去模糊问题都是高度病态的,直接求解无法做到.因此,通常的做法是通过抽取原始图像的已知统计先验信息进行正则化来帮助求解问题.两种常用的正则化是低秩和全变分.早期的相关工作单独使用这两种正则化.直到几年前,人们才考虑将这两种正则化结合起来.已有的结果表明,混合正则化模型比单一模型具有更好的性能.然而,目前的混合正则化方法只是采用凸方法,非凸的工作仍然是空白的.考虑到非凸正则化在很多种情况下都比凸正则化的效果要好,因此本文使用L1/2范数和Schatten-1/2范数提出了一种新的非凸混合模型.我们使用这两个非凸函数,因为它们的近端算子很容易计算.这种非凸混合正则化模型本质上是一个非凸线性约束问题,可以通过交替方向乘子法求解.然而,非凸性使得交替方向乘子法收敛十分困难.因此,我们转向求解原问题的惩罚问题.将交替最小化方法应用于惩罚问题就可以得到提出的算法,其中每个子步骤只涉及非常简单的计算.由于惩罚参数很大时,交替极小化算法速度会很慢,为了加速算法,针对惩罚参数我们使用了预热技术,即选取很小的初值但是在迭代过程中不断将参数增大.我们证明了该算法的收敛性.数值实验验证了本文提出的模型和算法的有效性.在非常温和的假设下,我们证明了算法的收敛性.数值实验验证了本文提出的模型和算法的有效性.     

A Fast Heuristic Global Learning Algorithm for Multilayer Neural Networks

This paper presents a novel Heuristic Global Learning (HER-GBL) algorithm for multilayer neural networks. The algorithm is based upon the least squares method to maintain the fast convergence speed, and the penalized optimization to solve the problem of local minima. The penalty term, defined as a Gaussian-type function of the weight, is to provide an uphill force to escape from local minima. As a result, the training performance is dramatically improved. The proposed HER-GBL algorithm yields excellent results in terms of convergence speed, avoidance of local minima and quality of solution.     

Penalty finite element method for the Navier-Stokes equations

We present an analysis of a penalty formulation of the stationary Navier-Stokes equations for an incompressible fluid. Subject to restrictions on the viscosity and prescribed body force, it is shown that there exists a unique solution to this penalty problem. The solution to the penalty problem is shown to converge to the solution of the Navier-Stokes problem as O(ε) where ε → 0 is the penalty parameter.Existence, uniqueness and stability properties for the approximate problem are then developed and we derive estimates for finite element approximation of the penalized Navier-Stokes problem presented here. Numerical studies are conducted to examine rates of convergence and sample numerical results presented for test cases.     

High-order time stepping scheme for pricing American option under Bates model

Pricing of European and American options under Bates model give rise to a partial integro-differential equation. In this paper a strongly stable fourth-order implicit predictor–corrector time stepping method based on exponential time differencing) is proposed for solving such problems. We provide stability, and convergence of the proposed method, and study the impact of the jump intensity, penalty and other parameters on convergence and solution accuracy. The American option constraint is enforced by using a penalty method. Spatial derivatives are approximated using second-order finite central differences which leads to block tridiagonal systems. The integral term is evaluated using simple quadrature where the non-locality of the jump term in such models leads to dense matrix. We treat the approximated integral term and nonlinear penalty term explicitly in time. Numerical experiments are demonstrated by discussing the efficiency, accuracy and reliability of the proposed method.     

Isoparametric C 0 interior penalty methods for plate bending problems on smooth domains

In this paper we develop isoparametric C ⁰ interior penalty methods for plate bending problems on smooth domains. The orders of convergence of these methods are shown to be optimal in the energy norm. We also consider the convergence of these methods in lower order Sobolev norms and discuss subparametric C ⁰ interior penalty methods. Numerical results that illustrate the performance of these methods are presented.     

Penalty functions and singular arcs in optimal steel refining

In a recent paper by Woodside et al. [1] the introduction of a penalty function to handle state constraints was found to lead to a singular arc in a problem of optimizing an electric steel refining process. Some numerical results were also given indicating a slow convergence of the penalty function solution to the solution of a constrained problem. In this correspondence a slightly different penalty function is used for the same problem. It is shown that the penalty function solution not only roughly approximates the solution to the state constrained problem, but that the singular arc and state constrained arcs coincide in the limit.     

A least-squares/penalty method for distributed optimal control problems for Stokes equations

The purpose of this paper is to construct an unconstrained optimal control problem by using a least-squares approach for the constrained distributed optimal control problem associated with incompressible Stokes equations. The constrained equations are reformulated to the equivalent first-order system by introducing vorticity, and then the least-squares functional corresponding to the system is enforced via a penalty term to the objective functional. The existence of a solution of the unconstrained optimal control problem is proved, and the convergence of this solution to that of unpenalized one is demonstrated as the penalty parameter tends to zero. Finite element approximations with error estimates are studied, and the relevant computational experiments are presented.     

An exact penalty function algorithm for control problems with state and control constraints

This paper describes an exact penalty function algorithm for solving control problems with state, control, and terminal constraints and establishes its convergence properties. A convex optimal control problem is defined whose solution yields a search direction which satisfies the control constraints and reduces a first-order estimate of the exact penalty function. Step length is determined using an Armijo-like procedure. An adaptive procedure for adjusting the penalty parameter completes the algorithm.     

A penalty PALM method for sparse portfolio selection problems

In this paper, we propose a penalty proximal alternating linearized minimization method for the large-scale sparse portfolio problems in which a sequence of penalty subproblems are solved by utilizing the proximal alternating linearized minimization framework and sparse projection techniques. For exploiting the structure of the problems and reducing the computation complexity, each penalty subproblem is solved by alternately solving two projection problems. The global convergence of the method to a Karush-Kuhn-Tucker point or a local minimizer of the problem can be proved under the characteristic of the problem. The computational results with practical problems demonstrate that our method can find the suboptimal solutions of the problems efficiently and is competitive with some other local solution methods.