Can the staircase of a deep linear network's loss be read backwards, as a measurement of a chosen spectrum?

Status: open · laid 2026-09-27 · type: lead

Statement. Train a deep linear network — a product of weight matrices, no nonlinearity — by gradient descent on a regression task, from a small initialisation. The loss does not fall smoothly. It falls in a staircase: the modes of the input–output correlation are learned one at a time, largest singular value first, each in a sudden drop, with a plateau between drops on which almost nothing happens. For the two-layer case the dynamics of each mode decouples and integrates in closed form; the time at which the mode of singular value σ is learned, from initialisation scale u₀, is

t(σ) ≈ ln(σ / u₀) / (2σ)

so the plateau before a mode is a function of that mode's singular value alone, and the ordering of the steps is the ordering of the spectrum. This much is known and solved (Saxe–McClelland–Ganguli's exact solutions for deep linear networks); it is the starting point, not the question.

The probe is the inverse problem. The formula is written as a forward map: a spectrum predicts a sequence of step times. Read it backwards and the step times are a measurement of the spectrum — and a strange kind of measurement, because it is made by watching an optimiser rather than by diagonalising anything. Two questions, in order:

1. Can step times read out a chosen spectrum? Construct data whose input–output correlation has singular values you pick — a geometric ladder, a two-cluster set, a spectrum with a gap — train, and recover the σ's from the measured step times against t(σ) with u₀ known. The quantities to report are the recovered σ's, their error, and the range of σ over which the readout works at all: the formula says t → ∞ as σ → 0, so the small modes are never learned and the readout must have a floor, set by the run length and by u₀.
2. What happens when two singular values nearly coincide? As σ₂/σ₁ → 1 the two predicted step times merge. Do the two drops merge smoothly into one drop of twice the depth, or is there a resolution limit — a minimum separation below which the staircase cannot distinguish two modes from one, and which would behave like an uncertainty relation between plateau length and spectral separation? The decoupling that makes the closed form work is a statement about the modes, not about the steps being separable in time, and an exactly degenerate pair is a rotational symmetry in that eigenspace: there is a candidate mechanism for the limit and the answer is not implied by the formula.

Why a probe and not a case. There are no competing branches yet, so there is nothing to discriminate. It is one clean question with a closed-form prior, a cheap test — a deep linear network on synthetic data, seconds per run — and no prediction on file.

No prediction and no falsifier are filed here. Both are the human's to write (MANUAL.md, Decisions only you make: a well-formed criterion written by a chat is not your prediction just because it is well-formed). If this becomes a run, the plateau-length formula above is the object the prediction would be written against.

Cross-refs. Sibling probes from the same session: P-028 (edge of stability, and whether its cascade is Feigenbaum's) and P-029 (double descent, and whether the spike is the Marchenko–Pastur edge). The run that opened the track is lab/runs/037-grokking-multiplicative-characters.md, whose statistic is also a spectral reading of a trained network's weights — there by DFT of an embedding, here by the clock.