Normalized LoRA: Fixing the Down-Projection Is Enough

Pretraining, supervised finetuning, and reinforcement learning all lean on the same adapter. Normalized Low-Rank Adaptation from Spherelab (Jiale Kang, Ziyin Yue, Zheng Zhan, Yangyi Huang, Weiyang Liu) asks whether LoRA’s training dynamics can be repaired by regularizing a single matrix rather than by adding machinery around it.

The paper’s landing page is at huggingface.co/papers/2608.31036 and the report identifier is arXiv:2608.31036v1 [cs.LG], 2026-08-31. No code repository is listed in the paper; treat the recipe below as directly reproducible from the described normalization.

Finding

Standard LoRA parameterizes an update as ΔW = α B A with B = 0 at initialization and A random. The consequence is immediate from the gradients at t = 0:

• ∂L/∂B = α G Aᵀ — non-zero, but gated entirely by the random A
• ∂L/∂A = 0 — A receives no gradient while B is still zero

Early optimization is governed by A(0) alone. If A has unbalanced column norms, different input coordinates project into the r-dimensional latent space at different scales. The paper shows the merged-weight step at initialization is ΔW = −η G P with P = α² AᵀA. P is a rank-r preconditioner on input coordinates; its diagonal entries α²‖aⱼ‖² act as per-coordinate learning rates. Under standard Kaiming or Gaussian initialization with variance ∝ 1/k, E[‖aⱼ‖²] ∝ r/k ≪ 1, so every coordinate learns at a fraction of the full-finetuning rate and with O(1/√r) random spread.

Normalized LoRA (NoRA) normalizes each column of A ∈ R^{r×k} along the rank dimension:

Norm(A) = [a₁/max(‖a₁‖,ε), … , aₖ/max(‖aₖ‖,ε)],  ‖Norm(A):,j‖ = 1
Δy = α B Norm(A) x

Norm(·) depends only on parameters, not on x, so the update remains linear and merges exactly as W₀ + B Norm(A) — unlike B Norm(Ax), which is input-dependent and non-mergeable (the MLA-style latent normalization that motivated the work).

Two forms are evaluated:

• NoRA — renormalize A along the rank dimension on every forward pass through training.
• NoRA-init — normalize A once at initialization, then train as standard LoRA. A deterministic variant, Block Identity Matrix Initialization (BIMI), uses A = [I_r, I_r, …, E_q] so every column is a unit basis vector by construction — also ‖aⱼ‖ = 1 without randomness.

With ‖aⱼ‖ = 1 and α = 1 (equivalently α = r under the α/r scaling convention used in the experiments), Diag(P) = I deterministically and E[P] = I over random directions. The adapter’s expected gradient norm aligns with full finetuning independent of r — the flat NoRA curve in the paper’s Figure 1, versus LoRA’s small and rank-dependent gradient norm.

Meaning

What changed is a conditioning claim about LoRA that is easy to miss behind aggregate scores: rank alone does not explain optimization collapse.

• The mechanism shown is diagonal gain, not subspace shape. Row-normalization (Norm_k) leaves Diag(P) random at scale r/k and does not help (Table 3). Random column-normalization and BIMI — same diagonal through different crosstalk patterns AᵀA off-diagonals — perform similarly. The controlled variable is Diag(P), not the specific directions in A.
• Initialization carries most of the effect; persistence adds stability. In SFT on Llama-3.2-3B, LoRA averages 37.93 across GSM8K/Math/HumanEval/MBPP. NoRA-init reaches 42.38 and NoRA reaches 43.37 (Table 5). The init-only fix captures most of the gain; continued normalization yields the remainder and a more stable trajectory.
• The generality test is DoRA. The same rank-dimension normalization applied to weight-decomposed DoRA moves DoRA from 38.30 to 41.40 under NoRA-init — evidence the principle is not tied to one parameterization.
• Retention matters. NoRA’s average drift on retained knowledge (MMLU, AGIEval, ARC-C) is +0.02, versus −0.56 for LoRA and −0.70 for MiSS under the same SFT (Table 5). Adaptation improved without amplifying forgetting.
• The pretraining stress test is MHA. Parameterizing all q/k/v/o projections at r = 64 under MHA on FineWeb-10BT, standard low-rank collapses on LAMBADA with diminished gradient norms; NoRA-init preserves gradient scale and avoids collapse (Section 4.2, Table 4). The authors connect this to the capacity-limited view in Schulman & Lab (2025): under the same rank budget, initialization geometry changes whether the trajectory can use that capacity at all.
• RLVR stability diverges by initialization family. On DeepSeek-R1-Distill-Qwen-1.5B with DAPO on DAPO-Math-17K, NoRA moves the overall RLVR average from 42.8 (LoRA) to 44.4, while PiSSA and MiLoRA — which rely on SVD of the pretrained weight — degrade sharply in RL (Table 6, consistent with Yin et al. 2025). NoRA needs no SVD, no activation statistics, and no extra parameters.

Costs are unchanged: no additional trainable parameters, no inference-time compute, and exact mergeability preserved. That is the design constraint the paper enforces — the MLA-inspired normalization Norm(Ax) would have improved conditioning but at the price of a nonlinear, non-mergeable adapter. NoRA moves the normalization from the feature to the projection to keep linearity.

Boundary conditions remain normal for early work of this kind: evaluation is on the Spherelab-controlled stack (FineWeb-10BT pretraining for 20,480 steps at 2K context and batch 256 with AdamW lr 3e-4; SFT on Llama-3.2-3B for 1 epoch at rank 32, α = 32, lr 2e-5, batch 128 per Appendix B; RLVR on a single 1.5B reasoning-distill for 1,024 steps at rank 32, α = 64). Scale to the 70B range, other families, and independent reproduction will determine how far the Diag(P) = I calibration generalizes. The implementation weight is in the single scaling choice α = r; deviating upward fits training loss more tightly but is reported to increase forgetting, so the default α:r = 1:1 is a stability–plasticity tradeoff, not a free gain.

Connection

The useful frame is preconditioning, not “better initialization.”

LoRA at initialization is full finetuning with its gradient right-multiplied by P = α²AᵀA — a low-rank, input-side preconditioner determined by A. A preconditioner that is too small and randomly stretched in every coordinate does not accelerate optimization; it attenuates and distorts it. Normalizing the column norms sets every diagonal entry of the preconditioner to one — unit gain per input coordinate — without prescribing which r directions the latent space must use.

The same correction appears in different costumes wherever a product of two factors must carry learning signal early. Weight normalization in deep networks (Salimans & Kingma, 2016; Liu et al., 2017, 2018) decouples length from direction so scale cannot drift into saturation. MLA normalizes the latent representation Ax before the up-projection to stabilize the bottleneck. NoRA applies the analogous decoupling to the adapter itself: length is fixed at one, direction remains trainable, and the update stays mergeable.

Spherelab’s phrasing is restrained and accurate to the evidence: rank-dimension normalization is a broadly applicable design dimension for low-rank adaptation, complementary to rank allocation, quantization, or weight decomposition. As a drop-in line — replace A initialization with Norm(A) — it reads as one of those small corrections that, if it survives replication, becomes standard because the cost of not applying it is a randomly misscaled first thousand steps.

Primitive implementation. Correct vector.

---

Authors: Jiale Kang, Ziyin Yue, Zheng Zhan, Yangyi Huang, Weiyang Liu (Yuanshi Intelligence, Microsoft Research, CUHK, Shenzhen Loop Area Institute). Paper: Normalized Low-Rank Adaptation, arXiv:2608.31036v1 [cs.LG], 31 Aug 2026. Venue: arXiv preprint. Code: not listed in the paper; method is huggingface.co/papers/2608.31036.