Geometry of optimisation

A neural network can admit many solutions which fit the training data but differ in their geometric and statistical character. Which solution is found depends on the learning algorithm. I study the dynamics of these algorithms and their bias towards particular solutions, with an emphasis on gradient descent. Two questions guide this work: how architectural choices create the conditions needed for convergence, and what happens when the step size is larger than classical theory permits, as is often the case in practice.

Gradient descent (GD) attempts to minimise a smooth loss function \(\ell:\mathbb R^p\to\mathbb R\) by taking steps in the direction of its negative gradient. For a neural network \(\phi_w\) with parameters \(w\in\mathbb R^p\), input-output training data \((a_i,b_i)_{i=1}^N\) determine a parameter-function map \(f(w)=(\phi_w(a_i))_{i=1}^N\). A cost function \(C\), such as the square or cross-entropy cost, compares the outputs of this map with the target outputs \((b_i)_{i=1}^N\). The loss and GD map are then

\[ \ell(w)=\gamma(f(w))=\frac1N\sum_{i=1}^N C(\phi_w(a_i),b_i), \qquad \mathrm{GD}(\eta,w):=w-\eta\nabla\ell(w), \]

where \(\eta>0\) is the step size. I use sharpness to mean the largest eigenvalue \(\lambda_1\) of the loss Hessian \(\nabla^2\ell\).

Architectures and convergence

A skip connection adds the input of a learned transformation to its output, for example \(z\mapsto z+g_w(z)\), providing a direct path across one or more layers. A normalisation layer rescales weights or intermediate outputs using their norms or statistics such as means and variances; weight normalisation applies this rescaling to weight vectors. Skip connections and normalisation layers are ubiquitous in practical neural net architectures, yet frequently neglected in theoretical optimisation analysis. The question of how these architectural choices affect optimisation was addressed in my 2023 paper with Jack Valmadre, Hemanth Saratchandran and Simon Lucey

Classical convergence results for GD can be obtained by assuming a Polyak–Łojasiewicz (PL) inequality and Lipschitz gradients:

\[ \|\nabla\ell(w)\|^2\geq\mu(\ell(w)-\ell_*), \qquad \|\nabla\ell(w)-\nabla\ell(\widetilde w)\|\leq\beta\|w-\widetilde w\|. \]

Here \(\ell_*\) is the infimum of the loss and \(\mu,\beta>0\). For sufficiently small step size, these properties give linear convergence of the loss to \(\ell_*\).

A sufficient condition to ensure the PL inequality (at least pointwise) is that \(f\) is a submersion, i.e. that \(Df(w)\) has full row rank for all \(w\). Overparametrisation makes this possible, since there must be at least as many parameters as output coordinates on the training data. A PL inequality for the cost then passes to the loss with a factor given by the smallest eigenvalue of \(Df(w)Df(w)^T\). Under certain conditions we give bounds on this eigenvalue in terms of the constituent layers, and demonstrate empirically that skip connections improve the relevant singular value distributions.

Three layer-Jacobian singular-value histograms shift to the right as skip connections are added; a fourth plot shows faster loss decrease
Adding skip connections shifts the singular value distributions of composite layer Jacobians upwards, with faster training in these experiments. Chain, Res and ResAvg are the chain network, residual network and modified residual network described in the paper. Means over ten trials. Figure 2.

Normalisation layers address the Lipschitz gradients condition. Bounds on the layers and their derivatives give Lipschitz gradients over loss sublevel sets throughout parameter space. We use this together with submersivity to prove global convergence for a class of weight- and entry-normalised residual networks.

Theorem (global convergence)

For certain weight-normalised residual networks, with linearly independent input data, appropriately rescaled residual branches and a continuously differentiable convex cost, there is a fixed step size for which GD converges in loss to the global infimum from any initialisation. This includes costs such as cross-entropy, whose infimum may only be attained at infinity.

A uniform PL constant is unavailable in this global setting. The proof instead shows that the positive regularity bounds along the trajectory deteriorate slowly enough for convergence to persist. The theorem has restrictive architectural and data hypotheses, but the mechanism it identifies also suggests changes to practical architectures. In particular, adding a suitably normalised deterministic map to learned convolutional skip connections improves training speed as predicted.

Training loss for original and modified ResNets on CIFAR10, CIFAR100 and ImageNet; the modified networks have lower loss
Original and modified ResNets, with learned convolutional skip maps \(A\) replaced by \(I+A\), where \(I\) is a rescaled average-pooling map. The plots show mean training loss, with one standard deviation shaded, over ten trials on CIFAR and three on ImageNet. Figure 3.

Large-step GD for the square cost

The preceding theory requires the step size to be smaller than twice the reciprocal supremum of the sharpness, which is a sufficient condition for linear stability of the algorithm at any fixed point. The reason for this classical stability threshold is already visible for a quadratic loss \(\ell(w)=\tfrac12\lambda w^2\), with \(\lambda>0\). Its GD map is

\[w\mapsto (1-\eta\lambda)w.\]

Thus \(0<\eta<2/\lambda\) gives convergence to zero; at \(\eta=2/\lambda\) the iterates oscillate with constant amplitude; and for \(\eta>2/\lambda\) a nonzero initial point gives divergence.

GD on a quadratic, below, at and above the classical stability threshold. Animation from my 2026 talk.

In 2021, Cohen, Kaur, Li, Kolter and Talwalkar demonstrated systematically that neural networks are routinely trained beyond this pointwise stability threshold. The sharpness rises during training, then fluctuates near or above \(2/\eta\), while the loss continues to decrease over longer timescales. They called this the edge of stability. The changing curvature of a neural-network loss allows behaviour which a fixed quadratic cannot exhibit.

Cohen and colleagues' training-loss and sharpness plots for several step sizes, with square loss above and cross-entropy below
Training loss (left) and sharpness (centre and right) at several step sizes, for square loss (top) and cross-entropy (bottom). Dashed horizontal lines mark \(2/\eta\). Larger step sizes lead to smaller sharpness thresholds, and loss decrease continues despite oscillations and violations of the pointwise threshold. From Cohen et al., Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability, ICLR 2021; figure reproduced in my talk.

This suggests that large step size can itself bias GD towards flatter minima: increasing \(\eta\) lowers the sharpness compatible with local stability. Larger step size can also lead to better test performance in some settings. The following ResNet18 experiment illustrates this even when both step sizes eventually fit the training data.

ResNet18 training and test accuracy on CIFAR10 for step sizes 0.1 and 0.001: both reach full training accuracy, while 0.1 has higher test accuracy
ResNet18 on CIFAR10, from my 2026 talk. Dotted curves show training accuracy and solid curves test accuracy; shading shows standard deviations. Both step sizes reach full training accuracy, but the larger step size gives better test accuracy in these runs.

Scalar-output least squares

My 2025 paper with Hancheng Min, Leandro Palma, Salma Tarmoun, Ziqing Xu and René Vidal seeks to make the geometric mechanism behind the bias towards flat minima explicit. We consider overparametrised least squares with scalar output,

\[\ell(w)=\frac12(f(w)-\tau)^2,\qquad f:\mathbb R^p\to\mathbb R,\quad p>1.\]

When \(\tau\) is a regular value, the solutions form a smooth hypersurface \(M=f^{-1}\{\tau\}\). This \(M\) is an invariant manifold for \(\mathrm{GD}\), and the key idea of our analysis is to prove a normal form for large-step GD in a neighbourhood of this invariant manifold, i.e. a choice of coordinates in which the implicit dynamics are rendered explicit.

The coordinates we choose are, up to a minor reparametrisation, tubular neighbourhood coordinates: all points near \(M\), can be realised by a pair \((x,y)\in M\times\mathbb{R}\) as in the picture below.

The solution hyperbola of a two-layer scalar network, with a shaded tubular neighbourhood, base point x, unit normal nu(x), signed displacement y and the corresponding nearby parameter Phi(x,y)
Tubular coordinates for the two-layer scalar network, adapted from my 2026 talk. The black curve is the solution manifold \(M\), the red point is the flat minimum \(F\), and the shaded neighbourhood is the tubular neighbourhood. A base point \(x\in M\), unit normal \(\nu(x)\) and signed displacement \(y\) determine the nearby parameter \(w=\Phi(x,y)=x+y\nu(x)\).

We also assume that there exists in \(M\) an isolated minimiser \(F\in M\) of the sharpness \(\lambda|_M\), and that \(\lambda|_M\) is geodesically strongly convex in a neighbourhood of \(F\).

Theorem (normal form near a flat minimum)

Subject to additional assumptions in the paper and a reparametrisation of \(y\), in the coordinates \((x,y)\) in a neighbourhood of the flat minimum \(F\), \(\mathrm{GD} = (\mathrm{GD}_x,\mathrm{GD}_y)\) has the form:

\[ \begin{aligned} \operatorname{GD}_x(\eta,x,y)&=x-\zeta(\eta,x)y^2\nabla_M\lambda(x)+O\!\left(|y|^3d_M(x,F)\right),\\ \operatorname{GD}_y(\eta,x,y)&=(1-\eta\lambda(x))y+y^3+O(y^4). \end{aligned} \]
Here \(\zeta(\eta,x)\) is a positive \(C^1\) function, \(\nabla_M\) is the Riemannian gradient, and \(d_M(x,F)\) is geodesic distance to \(F\).

Thus, to leading order, the tangential component is Riemannian GD on sharpness along \(M\), with step size \(\zeta(\eta,x)y^2\) controlled by the normal oscillation. The normal component is a parametrised flip bifurcation. At \(x = F\), the latter classically admits three convergence behaviours for all \(\eta\) sufficiently close to \(2/\lambda_F\): exponential convergence to zero when \(\eta\lambda_F<2\); polynomial convergence to zero when \(\eta\lambda_F=2\); exponential convergence to a stable period-two orbit when \(\eta\lambda_F>2\). These flip dynamics thus act as an automatic step-schedule on the Riemannian GD on \(\lambda\) coming from the \(x\)-update, from which the following convergence theorems follow.

Theorem (convergence near a flat minimum)

For all initialisations sufficiently close to \(F\) with \(y\neq0\) and step sizes sufficiently close to \(2/\lambda_F\), the iterates \((x_t,y_t)\) of GD satisfy:

  • Subcritical, \(\eta<2/\lambda_F\): transient instability is overcome in finite time, followed by exponential convergence to a global loss minimum. For initial base points \(x_0\notin F\), the limit is suboptimally flat.
  • Critical, \(\eta=2/\lambda_F\): \(\Theta(t^{-1/2})\) convergence to the flat minimum.
  • Supercritical, \(\eta>2/\lambda_F\): exponential convergence to a stable period-two orbit across the flat minimum, with amplitude of order \(\sqrt{\eta\lambda_F-2}\).
The three regimes for \(\ell(w_1,w_2)=\tfrac12(w_1w_2-1)^2\), shown together in one animation. The black curve is the solution manifold and the red point is its flat minimum. The animation labels its sharpness \(\lambda_*\); in our notation this is \(\lambda_F=2\), giving \(\eta_*=1\). From left to right: convergence to a suboptimally flat minimum, slower convergence to the flat minimum, and convergence to a period-two orbit across it.

These results hold for deep scalar factorisation: \(f(w)=w_L\cdots w_1\), with arbitrary depth \(L\geq2\) and nonzero target. The two-layer example depicts the same mechanism in a setting where the trajectories can be drawn.

Vector-valued outputs and manifolds of flat minima

Extending this analysis beyond scalar output raises two conceptual problems. First, there are now several normal directions. A useful normal form requires a dimension reduction which separates the oscillating direction from the others. Second, flat minima need not be isolated. For problems such as matrix factorisation, they form manifolds, so geodesic strong convexity about a single flat minimum must be replaced by an appropriate condition along this manifold.

My 2026 paper with René Vidal resolves these problems for overparametrised least squares with vector-valued output,

\[\ell(w)=\frac12\|f(w)-\tau\|^2,\qquad f:\mathbb R^p\to\mathbb R^q,\quad p>q.\]

We continue to assume that \(\tau\) is a regular value in the range of \(f\), so that the solutions again form a smooth manifold \(M=f^{-1}\{\tau\}\), and the nontrivial Hessian eigenvectors are precisely the normal directions to \(M\). There are now precisely \(q\) such nontrivial eigendirections \(\nu_1,\dots,\nu_{q}\) with corresponding eigenvalues \(\lambda_1\geq\cdots\geq\lambda_q\). Anticipating a normal form that requires taking the gradient of \(\lambda_1\), we consider only points where \(\lambda_1\) is strictly separated from the other eigenvalues.

In general, the local minima of sharpness on \(M\) can themselves form a manifold \(F\subset M\). The condition replacing strong convexity of \(\lambda_1\) is then that \(\lambda_1\) is Morse–Bott along \(F\): its Riemannian Hessian is positive in directions normal to \(F\) within \(M\). A concrete example is deep matrix factorisation: under fairly light assumptions the flat minima are not only a manifold, but a fibre bundle over a product of spheres, with typical fibre being the solution manifold of a one-lower-dimensional matrix factorisation problem.

A circle base, a curved interval fibre and a surface illustrating the fibre bundle of flat matrix-factorisation minima
The fibre-bundle structure for two-layer, 2 × 2 matrix factorisation. The base is a circle; the middle and right pictures show connected components of the fibre and total space. This is an illustration of a bundle embedded in a higher-dimensional parameter space. Figure 2 of the paper.

In the scalar output case \(q=1\), our coordinates were defined via a tubular neighbourhood, with the normal direction being precisely the sole nontrivial Hessian eigendirection. In the vector ouput case \(q>1\), the loss Hessian admits \(q\) nontrivial eigenvalues, necessitating dimension reduction. This is achieved using an invariant manifold and foliation, whose existence is the topic of my paper on locally invariant manifolds and foliations.

Theorem (locally invariant manifolds and foliations)

Any point \(x_*\in F\) admits a locally GD-invariant centre manifold containing a neighbourhood of \(x\) in \(M\) that is tangent to the sum of \(TM\) and the top Hessian eigendirection, whose points parametrise the leaves of a locally GD-invariant stable foliation which are tangent to the other nontrivial Hessian eigendirections.

A cartoon of the centre manifold and stable foliation is given below.

Blue solution manifold M inside a red centre manifold, with green stable leaves and red and green Hessian eigenvector directions
The solution manifold \(M\) (blue), centre manifold (red) and stable foliation (green). The fields \(\nu_1\) and \(\nu_{2:q}\) are the leading and remaining normal eigendirections of the Hessian. In this low-dimensional cartoon, \(F\) would be a single point of \(M\).

The coordinates we choose are, up to a reparametrisation of \(y_1\), centre-manifold and stable-foliation coordinates: all points sufficiently near a point of \(F\) can be realised by a pair \((x,y)\in M\times\mathbb R^q\), with \(y=(y_1,y_{2:q})\). Here \((x,y_1)\) are coordinates on the centre manifold and \(y_{2:q}\) are coordinates along the stable leaves, as in the picture above. Contraction along these leaves reduces the analysis to the coupling between \(x\) and \(y_1\).

Theorem (normal form near a manifold of flat minima)

Subject to the additional assumptions in the paper, for all points \(x_*\in F\) and all \(\eta\) sufficiently close to \(2/\lambda_1|_F\), the coordinates \((x,y_1,y_{2:q})\) may be chosen such that \(\mathrm{GD}=(\mathrm{GD}_x,\mathrm{GD}_{y_1},\mathrm{GD}_{y_{2:q}})\) has the form:

\[ \begin{aligned} \operatorname{GD}_x(\eta,x,y_1)&=x-\zeta(\eta,x)y_1^2\nabla_M\lambda_1(x)+O\!\left(|y_1|^3d_M(x,F)\right),\\ \operatorname{GD}_{y_1}(\eta,x,y_1)&=(1-\eta\lambda_1(x))y_1+y_1^3+O(y_1^4),\\ \operatorname{GD}_{y_{2:q}}(\eta,x,y_1,y_{2:q})&=(I-\eta\Lambda_{2:q}(x))y_{2:q}+O\!\left(|y_1|\|y_{2:q}\|+\|y_{2:q}\|^2\right). \end{aligned} \]

Here \(\zeta(\eta,x)\) is a positive \(C^1\) function, \(\nabla_M\) is the Riemannian gradient, \(d_M(x,F)\) is geodesic distance to \(F\), and \(\Lambda_{2:q}(x)\) has eigenvalues \(\lambda_2(x),\ldots,\lambda_q(x)\). The coordinates are \(C^4\) in the normal variables and \(C^1\) in the base variable.

Notice the \(x,y_1\) component of the update is the same as in the \(q=1\), and in particular is completely independent of \(y_{2:q}\); this is thanks to the invariance of the centre manifold and stable foliation. The new coordinate \(y_{2:q}\) simply contracts to zero thanks to the spectral gap condition and the assumption that \(\eta\) is sufficiently close to \(2/\lambda_1|_F\). The convergence results are therefore essentially the same as in the \(q=1\) case.

Theorem (convergence near a manifold of flat minima)

Under the same hypotheses, for all initialisations sufficiently close to \(F\) with \(y_{1,0}\neq0\) and step sizes sufficiently close to \(2/\lambda_F\), the iterates \((x_t,y_t)\) of GD satisfy:

  • Subcritical, \(\eta<2/\lambda_F\): transient instability is overcome in finite time, followed by exponential convergence to a global loss minimum. For initial base points \(x_0\notin F\), the limit is suboptimally flat.
  • Critical, \(\eta=2/\lambda_F\): \(\Theta(t^{-1/2})\) convergence to a point of \(F\).
  • Supercritical, \(\eta>2/\lambda_F\): exponential convergence to a stable period-two orbit across \(F\), with amplitude of order \(\sqrt{\eta\lambda_F-2}\).

The following plots show these three regimes in deep matrix factorisation.

Subcritical normal distances decay exponentially while distances along M to F settle at positive values
Subcritical regime: \(\|y_t\|\) (left) and \(d_M(x_t,F)\) (right), for five trials of a three-layer, 2 × 2 matrix-factorisation problem. Initial instability is followed by exponential convergence to a suboptimally flat solution. Figure 3.
Critical normal and tangential distances decrease with the reference rate t to the minus one half
Critical regime: both \(\|y_t\|\) and \(d_M(x_t,F)\) decay at the predicted \(t^{-1/2}\) rate. Dotted lines give reference slopes. Figure 4.
Supercritical normal distances approach a nonzero oscillation amplitude while distances along M to F decay exponentially
Supercritical regime: the normal component approaches a nonzero oscillation while the base coordinate converges towards \(F\). The limiting period-two orbit need not have exactly symmetric endpoints in the original parameters. Figure 5.

Large-step GD for cross-entropy type costs

Convergence theorems for large-step GD with logistic losses were already proved by Jingfeng Wu, Peter Bartlett, Matus Telgarsky and Bin Yu, leveraging the asymptotically flat nature of of such loss landscapes on linearly separable data using optimisation-theoretic techniques. A natural question is whether such theorems can also be obtained from the geometric perspective of invariant manifolds and normal forms as used in our least-squares theory.

The key problem that must be overcome in performing such analysis is the identification of a suitable invariant reference manifold. In my 2026 paper with Salma Tarmoun, Konstantinos Emmanouilidis, Ziqing Xu and René Vidal, we consider for this manifold a line \(\mathcal R=\operatorname{span}\{v\}\), with \(\|v\|=1\); we require \(\nabla\ell(xv)\in\mathcal R\), so GD preserves this line. A simple example of an objective admitting such a line is given in the animation below; we also prove that cross-entropy regression on suitably symmetric data admits such a line, namely the max-margin line.

GD on the toy loss \(\ell(x,y)=e^{-x}y^2\), from the NeurIPS 2026 slides. Left: iterates oscillate across the blue invariant line \(y=0\). Right: the coordinates \(x_t,|y_t|\) and the transverse stability quantity \(2\eta e^{-x_t}\). As \(x_t\) advances, this quantity falls below 2 and the oscillations damp. Here \(y=0\) is a line of zero-loss minima; separable logistic regression instead has a decreasing loss along its reference line.

At a high level, our coordinates now follow a similar decomposition to what we saw in the square loss case: write \(w=xv+y\), where \(x\in\mathbb R\) is the coordinate along the line and \(y\) is the orthogonal component. Define the restricted loss and curvature operator by

\[h(x)=\ell(xv),\qquad H(x)=\nabla^2\ell(xv),\]
Theorem (normal form around an invariant reference line)

Suppose that the top eigenvalue \(\lambda_1\) of \(H\) is simple and that its top eigenvector is orthogonal to \(\mathcal R\). Then in the coordinates \(x,y\) one has \(\mathrm{GD}=(\mathrm{GD}_x,\mathrm{GD}_y)\) taking the form

\[ \begin{aligned} \operatorname{GD}_x(\eta,x,y)&=x-\eta h'(x)-\frac\eta2 y^T H'(x)y+O(\|y\|^3),\\ \operatorname{GD}_y(\eta,x,y)&=(I-\eta H(x))y+O(\|y\|^2). \end{aligned} \]

Assuming an exponential tail on the loss and curvature in the same direction along the invariant line, as we see in cross-entropy classification on symmetric data, enables the following three-phase convergence result.

Theorem (four-phase convergence)

For all initial conditions sufficiently close to \(\mathcal R\) and all step sizes sufficiently close to the initial stability threshold, the iterates of GD evolve according to the following three phases:

  1. Instability and transition: In finite time, the iterates enter a region wherein the step size is below the stability threshold.
  2. Linear contraction: For finite time, the iterates undergo exponential contraction in the orthogonal direction toward \(\mathcal R\).
  3. Polynomial contraction: Asymptotically, the orthogonal component tends to zero at a polynomial rate as the sharpness and loss both tend to zero at a polynomial rate.

The normal form explains why these phases occur. In the top eigendirection of \(H\) with sharpness \(\lambda_1\), the leading multiplier in the \(y\)-update is \(1-\eta\lambda_1\). If \(\eta\lambda_1>2\), its sign is negative and its magnitude exceeds one, producing growing oscillations. Meanwhile, \(h'(x)<0\) drives \(x\) forwards. Since \(H'(x)\preceq0\), the quadratic coupling in the \(x\)-update adds forward drift: larger transverse excursions accelerate the motion towards smaller curvature. The instability thus helps remove its own cause, yielding the first phase of convergence.

Once the reference sharpness falls below the stability threshold of the step size, \(1-\eta\lambda_1\) becomes uniformly contractive for as long as \(\lambda_1\) can be bounded below by some fixed positive constant, yielding the linear contraction phase. Asymptotically, \(\lambda_1\) goes to zero at a polynomial rate, causing the contraction multiplier \(1-\eta\lambda_1\) to go to 1 at a polynomial rate, hence leading to the polynomial contraction of the iterates toward \(\mathcal R\).

The magnitude of the orthogonal component in symmetric logistic regression against iteration: initial amplification, transition, linear contraction and polynomial decay
Symmetric logistic regression: the orthogonal component \(\|y_t\|\), labelled \(\|\theta_t^\perp\|\) in the paper, against iteration \(t\). Initial amplification gives way to nonexpansion, linear contraction and polynomial decay. Dashed envelopes bound the latter two phases. From Tarmoun et al., Figure 3, upper panel, with the shared iteration axis retained (CC BY 4.0).

Open questions

This geometric analysis of large-step gradient descent is local, requiring an invariant manifold about which to prove normal forms and convergence theorems. To make this geometric theory global therefore requires identifying invariant manifolds connecting practical initialisations to solution sets, along which normal forms could be derived to cover the entire training trajectory.

Further questions concern nonlinear architectures and the inclusion of stochasticity and momentum in the optimisation algorithms. A statistical theory connecting these dynamical mechanisms to generalisation is also desirable.

Papers and further reading

Further reading