Solving sampling and optimization problems via Tamed Langevin MCMC algorithms in the presence of super-linearities

Abstract

In this thesis, we investigate the problem of sampling from distributions with super-linearly growing log-gradient and the applications to the related optimization problems. This challenge is tackled in different scenarios using explicit numerical schemes which have their foundations on the concept of taming, which is a technique deployed to remedy the issues posed by coefficients with super-linear growth.

In the first part of the thesis, under assumptions imposed by neural network applications, such as non-convexity and log-gradient local Lipschitz continuity, explicit non-asymptotic bounds in Wasserstein distances for the non-convex sampling problem are produced, using a novel tamed variant of the vanilla Stochastic Gradient Langevin Dynamics (SGLD) algorithm. Applying these results, the excess risk optimization problem is controlled, showing that the new sampling algorithm can also work as an optimizer.

In the second part of the thesis, a new tamed variant of the Unadjusted Langevin Algorithm (ULA) is used to solve the sampling and optimization problem when the target measure satisfies an isoperimetric inequality and has a local Lispchitz log- gradient. Explicit non asymptotic results are derived for the sampling problem in 2-Wasserstein, total variation and relative entropy distances while non-asymptotic bounds are also provided for the related excess risk optimization problem. A novel theoretical framework is presented to further explore the isoprerimetric constant’s dependence on key parameters.

In the last part of the thesis, two new tamed variants of the Kinetic Langevin Monte Carlo are considered in the strongly convex and local Lispchtiz log-gradient setting. Non-asymptotic bounds in 2-Wasserstein distance between the processes generated by our algorithms and the target measure are derived and are later used to bound the respective excess risk optimization problem.