Quasi-Monte Carlo Sampling for Solving Partial Differential Equations by Deep Neural Networks

Solving partial differential equations in high dimensions by deep neural networks has brought significant attentions in recent years. In many scenarios, the loss function is defined as an integral over a high-dimensional domain. Monte-Carlo method, together with a deep neural network, is used to overcome the curse of dimensionality, while classical methods fail. Often, a neural network outperforms classical numerical methods in terms of both accuracy and efficiency. In this paper, we propose to use quasi-Monte Carlo sampling, instead of Monte-Carlo method to approximate the loss function. To demonstrate the idea, we conduct numerical experiments in the framework of deep Ritz method. For the same accuracy requirement, it is observed that quasi-Monte Carlo sampling reduces the size of training data set by more than two orders of magnitude compared to that of Monte-Carlo method. Under some assumptions, we can prove that quasi-Monte Carlo sampling together with the deep neural network generates a convergent series with rate proportional to the approximation accuracy of quasi-Monte Carlo method for numerical integration. Numerically the fitted convergence rate is a bit smaller, but the proposed approach always outperforms Monte Carlo method.     

A penalty method for a finite-dimensional obstacle problem with derivative constraints

We propose a power penalty method for an obstacle problem arising from the discretization of an infinite-dimensional optimization problem involving differential operators in both its objective function and constraints. In this method we approximate the mixed nonlinear complementarity problem (NCP) arising from the KKT conditions of the discretized problem by a nonlinear penalty equation. We then show the solution to the penalty equation converges exponentially to that of the mixed NCP. Numerical results will be presented to demonstrate the theoretical convergence rates of the method.     

Path relinking and GRG for artificial neural networks

Artificial neural networks (ANN) have been widely used for both classification and prediction. This paper is focused on the prediction problem in which an unknown function is approximated. ANNs can be viewed as models of real systems, built by tuning parameters known as weights. In training the net, the problem is to find the weights that optimize its performance (i.e., to minimize the error over the training set). Although the most popular method for training these networks is back propagation, other optimization methods such as tabu search or scatter search have been successfully applied to solve this problem. In this paper we propose a path relinking implementation to solve the neural network training problem. Our method uses GRG, a gradient-based local NLP solver, as an improvement phase, while previous approaches used simpler local optimizers. The experimentation shows that the proposed procedure can compete with the best-known algorithms in terms of solution quality, consuming a reasonable computational effort.     

Bundle-based decomposition for large-scale convex optimization: Error estimate and application to block-angular linear programs

Robinson has proposed the bundle-based decomposition algorithm to solve a class of structured large-scale convex optimization problems. In this method, the original problem is transformed (by dualization) to an unconstrained nonsmooth concave optimization problem which is in turn solved by using a modified bundle method. In this paper, we give a posteriori error estimates on the approximate primal optimal solution and on the duality gap. We describe implementation and present computational experience with a special case of this class of problems, namely, block-angular linear programming problems. We observe that the method is efficient in obtaining the approximate optimal solution and compares favorably with MINOS and an advanced implementation of the Dantzig—Wolfe decomposition method.     

PowerNet: Efficient Representations of Polynomials and Smooth Functions by Deep Neural Networks with Rectified Power Units

Deep neural network with rectified linear units (ReLU) is getting more and more popular recently. However, the derivatives of the function represented by a ReLU network are not continuous, which limit the usage of ReLU network to situations only when smoothness is not required. In this paper, we construct deep neural networks with rectified power units (RePU), which can give better approximations for smooth functions. Optimal algorithms are proposed to explicitly build neural networks with sparsely connected RePUs, which we call PowerNets, to represent polynomials with no approximation error. For general smooth functions, we first project the function to their polynomial approximations, then use the proposed algorithms to construct corresponding PowerNets. Thus, the error of best polynomial approximation provides an upper bound of the best RePU network approximation error. For smooth functions in higher dimensional Sobolev spaces, we use fast spectral transforms for tensor-product grid and sparse grid discretization to get polynomial approximations. Our constructive algorithms show clearly a close connection between spectral methods and deep neural networks: PowerNets with $n$ hidden layers can exactly represent polynomials up to degree $s^n$, where $s$ is the power of RePUs. The proposed PowerNets have potential applications in the situations where high-accuracy is desired or smoothness is required.     

Analytical solution of stochastic differential equation by multilayer perceptron neural network approximation of Fokker–Planck equation

The Fokker–Planck equation is a useful tool to analyze the transient probability density function of the states of a stochastic differential equation. In this paper, a multilayer perceptron neural network is utilized to approximate the solution of the Fokker–Planck equation. To use unconstrained optimization in neural network training, a special form of the trial solution is considered to satisfy the initial and boundary conditions. The weights of the neural network are calculated by Levenberg–Marquardt training algorithm with Bayesian regularization. Three practical examples demonstrate the efficiency of the proposed method.     

Extracting post-nonlinear signal with specific kurtosis range

Blind source extraction (BSE) is an important technique to extract a desired source from the mixed signals and the post-nonlinear (PNL) mixture is more realistic model in many situations. In this paper, we address the problem of extracting the source of interest from the PNL mixture. First, the prior knowledge about the desired source, such as its normalized kurtosis range, can be treated as a constraint and incorporated into the contrast function. Therefore, BSE from the PNL mixture can be formulated a constrained optimization problem. Second, the inverse of the unknown nonlinear function is approximated by the multi-layer perceptions (MLP) network because neural network can uniformly approximate any continuous function if there is sufficient number of neurons in the hidden layer. Finally, the source of interest can be extracted from the PNL mixture by minimizing the constrained optimization problem with standard gradient descent method. Extensive computer simulations and experiments demonstrate the validity of our algorithm.     

Properties and methods for finding the best rank-one approximation to higher-order tensors

The problem of finding the best rank-one approximation to higher-order tensors has extensive engineering and statistical applications. It is well-known that this problem is equivalent to a homogeneous polynomial optimization problem. In this paper, we study theoretical results and numerical methods of this problem, particularly focusing on the 4-th order symmetric tensor case. First, we reformulate the polynomial optimization problem to a matrix programming, and show the equivalence between these two problems. Then, we prove that there is no duality gap between the reformulation and its Lagrangian dual problem. Concerning the approaches to deal with the problem, we propose two relaxed models. The first one is a convex quadratic matrix optimization problem regularized by the nuclear norm, while the second one is a quadratic matrix programming regularized by a truncated nuclear norm, which is a D.C. function and therefore is nonconvex. To overcome the difficulty of solving this nonconvex problem, we approximate the nonconvex penalty by a convex term. We propose to use the proximal augmented Lagrangian method to solve these two relaxed models. In order to obtain a global solution, we propose an alternating least eigenvalue method after solving the relaxed models and prove its convergence. Numerical results presented in the last demonstrate, especially for nonpositive tensors, the effectiveness and efficiency of our proposed methods.     

Approximating Unknown Mappings: An Experimental Evaluation

Different methodologies have been introduced in recent years with the aim of approximating unknown functions. Basically, these methodologies are general frameworks for representing non-linear mappings from several input variables to several output variables. Research into this problem occurs in applied mathematics (multivariate function approximation), statistics (nonparametric multiple regression) and computer science (neural networks). However, since these methodologies have been proposed in different fields, most of the previous papers treat them in isolation, ignoring contributions in the other areas. In this paper we consider five well known approaches for function approximation. Specifically we target polynomial approximation, general additive models (Gam), local regression (Loess), multivariate additive regression splines (Mars) and artificial neural networks (Ann).Neural networks can be viewed as models of real systems, built by tuning parameters known as weights. In training the net, the problem is to find the weights that optimize its performance (i.e. to minimize the error over the training set). Although the most popular method for Ann training is back propagation, other optimization methods based on metaheuristics have recently been adapted to this problem, outperforming classical approaches. In this paper we propose a short term memory tabu search method, coupled with path relinking and BFGS (a gradient-based local NLP solver) to provide high quality solutions to this problem. The experimentation with 15 functions previously reported shows that a feed-forward neural network with one hidden layer, trained with our procedure, can compete with the best-known approximating methods. The experimental results also show the effectiveness of a new mechanism to avoid overfitting in neural network training.     

Mixed-integer minimax dynamic optimization for structure identification of glycerol metabolic network

Cell metabolism is a dynamic regulation process, in which its network structure and/or regulatory mechanisms can change constantly over time due to internal and external perturbations. This paper models glycerol metabolism in continuous fermentation as a nonlinear mixed-integer dynamic system by defining the time-varying metabolic network structure as an integer-valued function. To identify the dynamic network structure and kinetic parameters, we establish a mixed-integer minimax dynamic optimization problem with concentration robustness as its objective functional. By direct multiple shooting strategy and a decomposition approach consisting of convexification, relaxation and rounding strategy, the optimization problem is transformed into a large-scale approximate multistage parameter optimization problem. It is then solved using a competitive particle swarm optimization algorithm. We also show that the relaxation problem yields the best lower bound for the optimization problem, and its solution can be arbitrarily approximated by the solution obtained from rounding strategy. Numerical results indicate that the proposed mixed-integer dynamic system can better describe cellular self-regulation and response to intermediate metabolite inhibitions in continuous fermentation of glycerol. These numerical results show that the proposed numerical methods are effective in solving the large-scale mixed-integer dynamic optimization problems.     

Unidirectional synchronization for Hindmarsh–Rose neurons via robust adaptive sliding mode control

This paper presents an adaptive neural network (NN) based sliding mode control for unidirectional synchronization of Hindmarsh–Rose (HR) neurons in a master–slave configuration. We first give the dynamics of single HR neuron which may exhibit spike-burst chaotic behaviors. Then we formulate the problem of unidirectional synchronization control of two HR neurons and propose a NN based sliding mode controller. The controller consists of two simple radial basis function (RBF) NNs which are used to approximate the desired sliding mode controller and the uncertain nonlinear part of the error dynamical system, respectively. The control scheme is robust to the uncertainties such as approximate errors, ionic channel noise and external disturbances. The simulation results demonstrate the validity of the proposed control method.     

Evaluation of fuzzy regression models by fuzzy neural network

In this paper, a novel hybrid method based on fuzzy neural network for approximate fuzzy coefficients (parameters) of fuzzy linear and nonlinear regression models with fuzzy output and crisp inputs, is presented. Here a neural network is considered as a part of a large field called neural computing or soft computing. Moreover, in order to find the approximate parameters, a simple algorithm from the cost function of the fuzzy neural network is proposed. Finally, we illustrate our approach by some numerical examples.     

An approximation-based approach for fuzzy multi-period production planning problem with credibility objective

This paper develops a fuzzy multi-period production planning and sourcing problem with credibility objective, in which a manufacturer has a number of plants or subcontractors. According to the credibility service levels set by customers in advance, the manufacturer has to satisfy different product demands. In the proposed production problem, production cost, inventory cost and product demands are uncertain and characterized by fuzzy variables. The problem is to determine when and how many products are manufactured so as to maximize the credibility of the fuzzy costs not exceeding a given allowable invested capital, and this credibility can be regarded as the investment risk criteria in fuzzy decision systems. In the case when the fuzzy parameters are mutually independent gamma distributions, we can turn the service level constraints into their equivalent deterministic forms. However, in this situation the exact analytical expression for the credibility objective is unavailable, thus conventional optimization algorithms cannot be used to solve our production planning problems. To overcome this obstacle, we adopt an approximation scheme to compute the credibility objective, and deal with the convergence about the computational method. Furthermore, we develop two heuristic solution methods. The first is a combination of the approximation method and a particle swarm optimization (PSO) algorithm, and the second is a hybrid algorithm by integrating the approximation method, a neural network (NN), and the PSO algorithm. Finally, we consider one 6-product source, 6-period production planning problem, and compare the effectiveness of two algorithms via numerical experiments.     

A simplified recurrent neural network for pseudoconvex optimization subject to linear equality constraints

In this paper, the optimization techniques for solving pseudoconvex optimization problems are investigated. A simplified recurrent neural network is proposed according to the optimization problem. We prove that the optimal solution of the optimization problem is just the equilibrium point of the neural network, and vice versa if the equilibrium point satisfies the linear constraints. The proposed neural network is proven to be globally stable in the sense of Lyapunov and convergent to an exact optimal solution of the optimization problem. A numerical simulation is given to illustrate the global convergence of the neural network. Applications in business and chemistry are given to demonstrate the effectiveness of the neural network.     

A review of Hopfield neural networks for solving mathematical programming problems

The Hopfield neural network (HNN) is one major neural network (NN) for solving optimization or mathematical programming (MP) problems. The major advantage of HNN is in its structure can be realized on an electronic circuit, possibly on a VLSI (very large-scale integration) circuit, for an on-line solver with a parallel-distributed process. The structure of HNN utilizes three common methods, penalty functions, Lagrange multipliers, and primal and dual methods to construct an energy function. When the function reaches a steady state, an approximate solution of the problem is obtained. Under the classes of these methods, we further organize HNNs by three types of MP problems: linear, non-linear, and mixed-integer. The essentials of each method are also discussed in details. Some remarks for utilizing HNN and difficulties are then addressed for the benefit of successive investigations. Finally, conclusions are drawn and directions for future study are provided.     

应力和位移约束下连续体结构拓扑优化

同时考滤应力和位移约束的连续体结构拓扑优化问题,很难用现有的均匀方法或变密度方法等求解。主要困难在于难以建立应力和位移约束与拓扑设计变量间显式关系式;即使建立了这种关系,也由于优化问题规模过大,利用常规的数学规划方法难以求解。隋允康、杨德庆曾提出了基于独立连续拓扑变量及映射变换（ＩＣＭ）的桁架结构拓扑优化模型。本文在此基础上,建立了以重量为目标,考虑应力和位移约束的连续体结构拓扑优化模型,并推导出     

The optimization technique for solving a class of non-differentiable programming based on neural network method

In this paper, the optimization techniques for solving a class of non-differentiable optimization problems are investigated. The non-differentiable programming is transformed into an equivalent or approximating differentiable programming. Based on Karush–Kuhn–Tucker optimality conditions and projection method, a neural network model is constructed. The proposed neural network is proved to be globally stable in the sense of Lyapunov and can obtain an exact or approximating optimal solution of the original optimization problem. An example shows the effectiveness of the proposed optimization techniques.     

Stability Analysis of Gradient-Based Neural Networks for Optimization Problems

The paper introduces a new approach to analyze the stability of neural network models without using any Lyapunov function. With the new approach, we investigate the stability properties of the general gradient-based neural network model for optimization problems. Our discussion includes both isolated equilibrium points and connected equilibrium sets which could be unbounded. For a general optimization problem, if the objective function is bounded below and its gradient is Lipschitz continuous, we prove that (a) any trajectory of the gradient-based neural network converges to an equilibrium point, and (b) the Lyapunov stability is equivalent to the asymptotical stability in the gradient-based neural networks. For a convex optimization problem, under the same assumptions, we show that any trajectory of gradient-based neural networks will converge to an asymptotically stable equilibrium point of the neural networks. For a general nonlinear objective function, we propose a refined gradient-based neural network, whose trajectory with any arbitrary initial point will converge to an equilibrium point, which satisfies the second order necessary optimality conditions for optimization problems. Promising simulation results of a refined gradient-based neural network on some problems are also reported.     

A capable neural network model for solving the maximum flow problem

This paper presents an optimization technique for solving a maximum flow problem arising in widespread applications in a variety of settings. On the basis of the Karush–Kuhn–Tucker (KKT) optimality conditions, a neural network model is constructed. The equilibrium point of the proposed neural network is then proved to be equivalent to the optimal solution of the original problem. It is also shown that the proposed neural network model is stable in the sense of Lyapunov and it is globally convergent to an exact optimal solution of the maximum flow problem. Several illustrative examples are provided to show the feasibility and the efficiency of the proposed method in this paper.     

Modelling and control in pseudoplate problem with discontinuous thickness

This paper concerns an obstacle control problem for an elastic (homogeneous) and isotropic) pseudoplate. The state problem is modelled by a coercive variational inequality, where control variable enters the coefficients of the linear operator. Here, the role of control variable is played by the thickness of the pseudoplate which need not belong to the set of continuous functions. Since in general problems of control in coefficients have no optimal solution, a class of the extended optimal control is introduced. Taking into account the results of G-convergence theory, we prove the existence of an optimal solution of extended control problem. Moreover, approximate optimization problem is introduced, making use of the finite element method. The solvability of the approximate problem is proved on the basis of a general theorem. When the mesh size tends to zero, a subsequence of any sequence of approximate solutions converges uniformly to a solution of the continuous problem.