Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

github.com

Post reply on HN
Print view View original

31–40 of 100 posts

Re: Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

#31

Post by kgeist »

Heh, for a couple last days, I've been doing this exact kind of "neuroanatomy" on Qwen2.5/Qwen3 too. Fascinating stuff. To make it easier to fiddle with the network, I created a small inference engine that is stripped of all the framework magic, just raw matmuls and all (main inference loop is just 50 lines of code!). For example, it's trivial to remove a layer: i just skip it in code with a simple "if". I've found that removing some layers doesn't appear to change anything (based on the vibes at least). If you remove some later layers, the model forgets how to insert the EOS token and keeps chatting ad finitum (still coherently). Removing earliest layers makes the model generate random garbage. Turns out abliteration is not hard to do, 10 examples was enough to find the refusal vector and cancel most refusals. Interestingly, I've found that refusal happens in the middle layers too (I think, layer 12 out of 26)

From what I understand, transformers are resistant to network corruption (without complete collapse) thanks to residual connections.

I tried to repeat some layers too but got garbage results. I guess I need to automate finding the reasoning layers too, instead of just guessing.

Re: Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

#33

Post by 3eb7988a1663 »

m3kw9 wrote: post #28

What, just randomly choose some "layer" and duplicate it and give some arbitrary reasoning went from 0.2 -> 0.7, i don't know man. You need to use real benchmarks.

Someone recently posted the exact same idea to much acclaim: https://news.ycombinator.com/item?id=47322887

Re: Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

#35

Post by kirill5pol »

4bpp wrote: post #27

Assuming the benchmarks are sound (rather than capturing a fluke), the provided explanation still does not pass the smell test. As far as I can tell, there is nothing about the training process of these models that would encourage them to make the output of any layer apart from (n-1) meaningful as the input of layer n, unless perhaps these layers were initialised as identity and the training process did not get to ch…

Basically all of them are using residual connections so it’s not that surprising honestly

Re: Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

#37

Post by abhikul0 »

xlayn wrote: post #23
Karuma wrote: post #15

Wow, every single word in the original post and on that README.md is pure LLM. How sad. In any case, this has been done at least since the very first public releases of Llama by Meta... It also works for image models. There are even a few ComfyUI nodes that let you pick layers to duplicate on the fly, so you can test as many as you want really quickly.

Fair point on the writing style, I used Claude extensively on this project, including drafting. The experiments and ideas are mine though. On the prior art: you're right that layer duplication has been explored before. What I think is new here is the systematic sweep toolkit + validation on standard benchmarks (lm-eval BBH, GSM8K, MBPP) showing exactly which 3 layers matter for which model. The Devstral logical deduc…

I only know of this one: https://github.com/shootthesound/comfyUI-Realtime-Lora. Haven't played with any layer manipulation though.

Re: Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

#39

Post by smusamashah »

xlayn wrote: post #18
SyzygyRhythm wrote: post #8

If running twice is good, then is running N times even better? I wonder if you could even loop until some kind of convergence, say hitting a fixed point (input equals output). I wonder if there's even a sort of bifurcation property where it sometimes loops A->A->A, but other times A->B->A, or more, rather like the logistic map fractal.

I explored that, again with Devstral, but the execution with 4 times the same circuit lead to less score on the tests. I chat with the model to see if the thing was still working and seemed coherent to me, I didn't notice anything off. I need to automate testing like that, where you pick the local maxima and then iterate over that picking layers to see if it's actually better, and then leave the thing running overnig…

Can Karpathy's autoresearch be used on this to explore what works and what does not? That is supposed to automate research like this from what I understand.

Re: Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

#40

Post by gmerc »

kgeist wrote: post #31

Heh, for a couple last days, I've been doing this exact kind of "neuroanatomy" on Qwen2.5/Qwen3 too. Fascinating stuff. To make it easier to fiddle with the network, I created a small inference engine that is stripped of all the framework magic, just raw matmuls and all (main inference loop is just 50 lines of code!). For example, it's trivial to remove a layer: i just skip it in code with a simple "if". I've found t…

Hook it up in autoresearch?
Post reply on HN