MPS and CPU training diverge for the same seed in R26_training_dynamics — so a retrain meant to reproduce 037's W_E must run on the same device, and be checked

MPS and CPU training diverge for the same seed

The claim. In R26_training_dynamics, two runs with the same seed, same hyperparameters, same data split and the same script do not follow the same trajectory on MPS and on CPU. They start from what is effectively the same initialisation and separate within a hundred steps.

Evidence, from run 037's own artifacts

The record already contains the comparison; it was made for timing, not for this. The pair is R26_training_dynamics/metrics/add-transformer-s0-size100-mps.json and …-size100-cpu.json — the 300-step sizing runs of run 037's control arm, task add_mod_113, seed 0. Their hyper blocks are identical, both have n_train 3,831 and n_test 8,938, and both carry time_only: true, so neither saved an embedding.

At step 0 the two agree to about 10⁻⁶ — the same initialisation, evaluated in different kernel orders. By step 100 they are three orders of magnitude further apart than that, and the gap is accumulated arithmetic, not a different model. The narrowing at step 300 is not convergence back together: both losses are collapsing toward zero, so the absolute gap shrinks while the runs stay distinct.

The consequence, which is operational

Run 037 saved only W_E (R26_training_dynamics/emb/*.npy); there is no checkpoint. So any attempt to read 037's attention heads, MLP or W_L has to retrain, and then:

• the retrain must run on the same device as the original — MPS, for every seed of 037; and
• the reproduction of the saved W_E must be checked, seed for seed, before anything downstream is read. Not assumed. A mismatch in a head is otherwise indistinguishable from a mismatch in the model.

This is why courses/src/grokking-from-characters.md §3.7 states the reproduction check as a prerequisite rather than a formality.

What is not known, and is not claimed here

• Same-device determinism on MPS is untested. This note shows MPS ≠ CPU. It does not show MPS = MPS. PyTorch does not guarantee bit-identical results across runs on the MPS backend, and nothing in this repo has checked it. The retrain prerequisite above is therefore necessary and possibly not sufficient.
• Whether the divergence matters at run scale is unknown. The evidence is 300 steps. Run 037's launches were 25,000 steps and passed through a grokking transition — train accuracy at 0.99 by step 200–300, test accuracy crossing much later. A difference of 10⁻³ early in a run that then reorganises itself may be irrelevant, or may select a different set of key frequencies, since which frequencies a seed lands on already varies seed to seed (run 037's three control seeds share no top-five set, and Nanda et al.'s Appendix C.2.1 lists four runs of one architecture with four different key-frequency sets). Nothing here predicts which.

The cheap check that would settle the first gap

Run the same seed twice on MPS for 300 steps and compare the logs step by step — the same --time-only sizing command the pair above came from, run twice on one device instead of once on each. Identical logs mean same-device determinism holds at this length and the retrain prerequisite is sufficient; any difference means reproduction cannot be checked by equality at all and needs a tolerance, which is a different and larger question.

Not run here. It is recorded as the next cheap thing, not as a result.

Back-links (2026-09-29)

Dated pointers added by the codifier from lab/backlinks.py; nothing above this heading was changed.

• Run 037 — confirmed (2026-09-27, measured). lab/runs/037-grokking-multiplicative-characters.md: CONFIRMED 2026-09-28 by the repo owner, scored on the prediction's intent.