Abstract
The training of deep neural networks predominantly relies on a combination of gradient-based optimization and back-propagation for the computation of the gradient. While incredibly successful, this approach faces challenges such as vanishing or exploding gradients, difficulties with non-smooth activations, and an inherently sequential structure that limits parallelization. Lifted training methods offer an alternative by reformulating the nested optimization problem into a higher-dimensional, constrained optimization problem where the constraints are no longer enforced directly but instead penalized with penalty terms. This chapter introduces a unified framework that encapsulates various lifted training strategies—including the Method of Auxiliary Coordinates ( MAC ), Fenchel Lifted Networks , and Lifted Bregman Training —and demonstrates how diverse architectures, such as Multi-Layer Perceptrons ( MLPs ), Residual Neural Networks ( ResNets ), and Proximal Neural Networks ( PNNs ), fit within this structure. By leveraging tools from convex optimization, particularly Bregman distances, the framework facilitates distributed optimization, accommodates non-differentiable proximal activations, and can improve the conditioning of the training landscape. We discuss the implementation of these methods using block-coordinate descent ( BCD ) strategies—including deterministic implementations enhanced by accelerated (e.g., Nesterov, Heavyball) and adaptive (e.g., Adam) optimization techniques—as well as implicit stochastic gradient methods ( ISGM ). Furthermore, we explore the application of this framework to inverse problems, detailing methodologies for both the training of specialized networks (e.g., unrolled architectures) and the stable inversion of pre-trained networks. Numerical results on standard imaging tasks validate the effectiveness and stability of the lifted Bregman approach compared to conventional training, particularly for architectures employing proximal activations.
| Original language | English |
|---|---|
| Title of host publication | Handbook of Numerical Analysis |
| Subtitle of host publication | Machine Learning Solutions for Inverse Problems: Part B |
| Publisher | Elsevier |
| Chapter | 9 |
| Pages | 425-485 |
| Number of pages | 61 |
| Volume | 27 |
| ISBN (Print) | 9780443428173 |
| DOIs | |
| Publication status | Published - 2026 |
Keywords
- Algorithm unrolling
- Bregman distances
- Distributed optimization
- Inverse problems
- Lifted training
- Network inversion
- Proximal neural networks
ASJC Scopus subject areas
- Numerical Analysis
- Modelling and Simulation
- Computational Mathematics
- Applied Mathematics
Fingerprint
Dive into the research topics of 'A Unified Framework for Lifted Training and Inversion Approaches'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver