Skip to main navigation Skip to search Skip to main content

A Unified Framework for Lifted Training and Inversion Approaches

Research output: Chapter in Book/Report/Conference proceeding › Chapter

Abstract

The training of deep neural networks predominantly relies on a combination of gradient-based optimization and back-propagation for the computation of the gradient. While incredibly successful, this approach faces challenges such as vanishing or exploding gradients, difficulties with non-smooth activations, and an inherently sequential structure that limits parallelization. Lifted training methods offer an alternative by reformulating the nested optimization problem into a higher-dimensional, constrained optimization problem where the constraints are no longer enforced directly but instead penalized with penalty terms. This chapter introduces a unified framework that encapsulates various lifted training strategies—including the Method of Auxiliary Coordinates ( MAC ), Fenchel Lifted Networks , and Lifted Bregman Training —and demonstrates how diverse architectures, such as Multi-Layer Perceptrons ( MLPs ), Residual Neural Networks ( ResNets ), and Proximal Neural Networks ( PNNs ), fit within this structure. By leveraging tools from convex optimization, particularly Bregman distances, the framework facilitates distributed optimization, accommodates non-differentiable proximal activations, and can improve the conditioning of the training landscape. We discuss the implementation of these methods using block-coordinate descent ( BCD ) strategies—including deterministic implementations enhanced by accelerated (e.g., Nesterov, Heavyball) and adaptive (e.g., Adam) optimization techniques—as well as implicit stochastic gradient methods ( ISGM ). Furthermore, we explore the application of this framework to inverse problems, detailing methodologies for both the training of specialized networks (e.g., unrolled architectures) and the stable inversion of pre-trained networks. Numerical results on standard imaging tasks validate the effectiveness and stability of the lifted Bregman approach compared to conventional training, particularly for architectures employing proximal activations.

Original languageEnglish
Title of host publicationHandbook of Numerical Analysis
Subtitle of host publicationMachine Learning Solutions for Inverse Problems: Part B
PublisherElsevier
Chapter9
Pages425-485
Number of pages61
Volume27
ISBN (Print)9780443428173
DOIs
Publication statusPublished - 2026

Keywords

  • Algorithm unrolling
  • Bregman distances
  • Distributed optimization
  • Inverse problems
  • Lifted training
  • Network inversion
  • Proximal neural networks

ASJC Scopus subject areas

  • Numerical Analysis
  • Modelling and Simulation
  • Computational Mathematics
  • Applied Mathematics

Fingerprint

Dive into the research topics of 'A Unified Framework for Lifted Training and Inversion Approaches'. Together they form a unique fingerprint.

Cite this