Weak-To-Strong Warm-Starting NLAs: Preliminary Evidence

(if you are unfamiliar with NLA's, read the quick primer)

Problems with NLA's

(this section exists mainly to clarify a set of problems with NLA's, to then introduce how these concerns might be improved)

There are three shaping forces of NLA's:

• The initial warm-start supervised learning
• The autoencoder reconstruction loss
• The KL penalty of GRPO, keeping the steps within the warm-start distribution. (small)

All of these forces can be seen as proxies for "a human explanation of this vector", and in the worst case, will just become Goodharted with sufficient optimization pressure.

However, one might object that, similarly to RLHF, there is a large "solution space" that kind of looks okay, i.e. today's LLMs are fairly nice most of the time even in the absence of any sort of extremely sophisticated post-training procedure. Likewise, NLA's might one day be a tool that, you know, explains the activations of LLMs fairly well,¹1 Put differently, the aim is to figure out how well NLA's do on a scale of "just more GPUs" to "fundamentally flawed", with "some tweaks here and there" being my hopeful guess. with some weird edges.²2 How useful such a state is for alignment of superintelligence is an open question.

• By the same token, I am also more optimistic than Zhou & TurnTrout, of course it is important to note that NLA's can be made to produce implausible explanations quite easily, yet the same plausibly holds for CoT, i.e. you can disturb the legibility quite easily.
• It must still be noted here that RL did not repair the implausiblity significantly.

(Because today's open-weight NLA's don't seem that useful yet, we want to estimate how useful throwing more GPU's / RL on them might be. Three sources of evidence:

• Most of Anthropic's benchmarks of NLA performance improve with RL training. However, there are notable exceptions.
• How training Qwen (here) improves some set of NLA evaluations.
• If across-problem variance in reconstruction loss tracks holistic evaluations, which is then extrapolated to across-training variance in reconstruction.)

It is claimed that a broad set of holistic evaluations of NLA's improve with RL training.

In general, it has to be noted though "evaluating NLA's" is a pretty hard problem, we have to generalize from simple cases where we know "now it ought to be thinking XYZ" to "all of LLM thought".

A useful canary here is: Assuming the minds of LLM's are fairly alien, has interp extracted an abstraction that is just slightly different from a human one? Something that is just way easier to implement in transformer circuits compared to biological neurons?³3 This is vaguely related to Yudkowsky's prediction market, and I'm sure lots of smarter ink has been spilled on similar questions already.

That is, the NLA's should be like "to properly explain this, I'll need to define what an ajwkodwklwekqs is, otherwise you wouldn't get it"

Furthermore, warm-start is needed, i.e. pure RL produces illegible explanations with low loss, which is evidence that the selection force of RL is pointing in the wrong direction, at least when starting from a random point.

oakhu & greenblatt showed that out of the three forces, it is plausible warm-start is the strongest, with the RL just producing a few bits of optimization pressure (i.e. Sonnet best-of-N confabulates better thoughts)

What can we do about it?

• Some cases were we might know that LLM's are thinking XYZ:
  • Give them the output of an OEIS sequence (that they can predict, and cannot have memorized), and see if the NLA gives back the generating function.
    • In general, if we have a sequence of tokens f(1), f(2), .., f(N), base models ought to think of the most parsimonious function f.
  • The main failure mode of such things is mechanistically that the AV reads out the shallow tokens of the activation vector, reconstructs what the model "ought to be thinking" (cases in which the AV can be sure the model must be thinking XYZ based on context exactly counfound with cases where we can be sure the model must be thinking XYZ based on context)
  • We would need problems such that we can be sure the AV cannot predict the underlying sequence, but the model can. Then it must be reconstructing the model's deeper activations. That is, the AV must be kept stupid
• Keeping the AV stupid, shot #1: (SFT memorizes, RL Generalizes)
  • Train the AV on Best-of-N SFT instead of GRPO. The rough intuition is GRPO does intelligence, BoN-SFT does memorization
    • AV Intelligence is sketchy; it's the thing that leads to steganography, it's the thing that lets it confabulate what the LLM is thinking from shallow patterns, instead of deeply checking.
    • AV's translate from activation-neuralese to English, translating is mostly learning many vocabulary-pairs, it's memorization.
    • However, it is still an open question how to update the AR fairly with this, I currently don't see a way other than just updating it on every activation (i.e. best-of-N would obviously just update on it's best work), which means the whole thing is not much cheaper computationally.
• Keeping the AV stupid, shot #2: Weak-to-strong AV's
  • Base case: A weak model is so simple s.t. we can train down the loss fairly well, avoid these failure modes as follows:
    • The AV must reconstruct a very large part of the activation vector, meaning it must also catch the "deeper parts", i.e. what the model is really thinking
    • There is some complicating evidence, i.e., while it's a priori plausible that simpler models have simpler to-read-out thoughts, it is not clearly empirically true that NLAs read them out better as well.
  • Induction Case: There is a strong model, perhaps from the same training run, which is just an iota smarter, that thinks fairly similarly. We can warm-start it with the weak model's thoughts, and even if RL can just change a few bits, we are fairly likely to extrapolate correctly.
    • Unfortunately, the size of the extrapolation trades off against the compute required here.
    • Regarding the size of the extrapolation one could perform, this experiment shows different NLA explanations perform surprisingly well when implanted across models. One may either take this as evidence that there exists a platonic natural language explanation of model thought (which is what we would need), or as evidence that all models haven't moved off their warm-start distribution yet.
  • Shall we be successful, we can recursively train it up again.

To even make this possible, one would need to train an NLA on a model that is smaller than Qwen3-8B, hope it has decently low loss, then warm-start Qwen3-8B with its implanted NLA thoughts, in the hope that outperforms compute-matched baseline.

• However, a hurdle is to even have a base case, i.e., to have a weak model whose thoughts we understand well in the first place.
• Furthermore, Qwen2.5-7B is the N=20 undergrad online survey of AI Safety. That is, anything can be shown on Qwen3-8B.

From Redwood's Consistency Lens, the quote goes:

The structure of LLMs activation are close to the structure of short text. Some evidence for this is how well word2vec embeddings capture human intuitions about words, despite being simply trained at some language compression task.

That is, it might be sufficient to learn a mapping, in other words, there is a long list of particularities (i.e. directions in vector space are known to be exp in n, with little interference, see Olah). Therefore it ought to be sufficient to have something that is as smart as 'can map abstraction onto human ones, will put it into writing correctly'

• This may mean a very small model is enough, and the warm-start is equally questionable, because it injects a lot of intelligence where we don't need it.
• Rewriting a Turing Machine to Lambda calculus is constant overhead, so if the 'words' of the two languages are aggregated or combined in different ways (or recursively build up from different principles), I would still expect the translation to be fairly cheap, albeit a sketchy metaphor

Gao & Schulman and J. Hilton study reward model overoptimization, in which they compare RL to BoN-SFT. (There are however a number of opposite key desiderata for RLHF when compared to interp-alignment)⁴4 With the term interp-alignment I mean finding a selection force that produces legible human text. For example, the sparsity constraint of SAE, or the reconstruction force (both SAE & NLA's). In my eyes, it represents a promising area of study.

• RL is far less KL-efficient than BoN, meaning it "consumes" more KL, for the same amount of (over)optimization.

Initial experimental results

Per Gao et al, BoN only yields log(n) bits for n steps, with our limited compute we seldomly observe a difference from the release version of Qwen2.5-7B. I am currently unsure what to make of this.

Best-of-N captured 67% of GRPO's reward gain while drifting an order of magnitude less from the initial policy, i.e. best-of-N keeps things simple, no scary intelligence