Low-rank updates, plotted

LoRA freezes a weight matrix and learns a correction that is the product of two thin matrices. The rank of that product is the only real knob, and it is tempting to turn it up. This placeholder post sketches what turning it does, with synthetic numbers shaped like a real sweep.

The update as pixels

For a frozen weight $W_0 \in \mathbb{R}^{d \times k}$ , LoRA trains

\begin{gathered} W = W_0 + \frac{\alpha}{r} B A, \\ B \in \mathbb{R}^{d \times r},\quad A \in \mathbb{R}^{r \times k},\quad r \ll \min(d, k). \end{gathered}

$B$ starts at zero, so training begins exactly at the pretrained model.¹ Drag the rank and watch the product gain structure.

rank 4

trainable, one 4096² layer—

full fine-tune—

fraction—

Figure 1Random

B

and

A

and their product. Orange cells are positive, blue negative. At rank 1 every row of

\Delta W

is a multiple of the same vector; by rank 16 the pattern looks like noise.

Learning rate against rank

The useful question is not which rank wins but whether the best learning rate moves with it. With the $\alpha / r$ scaling, it mostly does not.² Hover the chart to read values; use the chips to hide or isolate a series.

Figure 2Final loss against learning rate, log scale. Synthetic data. The LoRA curves share one optimum near

4 \times 10^{-4}

; full fine-tuning prefers a learning rate about ten times smaller.

Training curves

Raw loss is noisy. An exponential moving average with factor $\beta$ shows the trend:

\bar{\ell}_t = \beta\, \bar{\ell}_{t-1} + (1 - \beta)\, \ell_t .

β 0.80

Figure 3Training loss over 2,000 steps. Synthetic data. Slide β to zero to see the raw noise.

Applying it

With Hugging Face PEFT the whole setup is a config:

lora.py

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.0,
)
model = get_peft_model(model, config)
model.print_trainable_parameters()

Targeting the MLP layers as well as attention matters more than the rank in most small sweeps. That claim is the subject of a real post, later.

Notes

$A$ is initialised randomly and $B$ at zero. The reverse also works, but one of the two must be zero for $\Delta W = 0$ at step one. ↩︎
Dividing by $r$ keeps the size of the update roughly constant as the rank changes, so the same learning rate keeps working. ↩︎

Cite this post

BibTeX

@article{sahu2026low,
  author = {Surya Kant Sahu},
  title  = {Low-rank updates, plotted},
  year   = {2026},
  url    = {https://ojus1.github.io/posts/low-rank-updates-plotted/}
}