Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

github.com

Post reply on HN
Print view View original

71–80 of 100 posts

Re: Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

#71

Post by Karuma »

abhikul0 wrote: post #37
xlayn wrote: post #23

Earlier quoted context omitted.

Fair point on the writing style, I used Claude extensively on this project, including drafting. The experiments and ideas are mine though. On the prior art: you're right that layer duplication has been explored before. What I think is new here is the systematic sweep toolkit + validation on standard benchmarks (lm-eval BBH, GSM8K, MBPP) showing exactly which 3 layers matter for which model. The Devstral logical deduc…

I only know of this one: https://github.com/shootthesound/comfyUI-Realtime-Lora . Haven't played with any layer manipulation though.

I was thinking more like this one: https://github.com/AdamNizol/ComfyUI-Anima-Enhancer/

"It adds the Anima Layer Replay Patcher, which can enhance fine detail and coherence by replaying selected internal blocks during denoising."

Re: Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

#72

Post by SCLeo »

4bpp wrote: post #54
zozbot234 wrote: post #52

Earlier quoted context omitted.

> far as I can tell, there is nothing about the training process of these models that would encourage them to make the output of any layer apart from (n-1) meaningful as the input of layer n Wouldn't "pass-through" identity connections have exactly that effect? These are quite common in transformer models.

Yeah, that's what I meant with "initialised as identity and the training process did not get to change them much".

There are explicit residual connections in a transformer block. Look up "residual connections" in Google images and you will see.

Re: Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

#73

Post by simgt »

regularfry wrote: post #67
yorwba wrote: post #51

Earlier quoted context omitted.

After reading both the original post and this submission, what do you think is new here?

> The weird part: different duplication patterns create different cognitive "modes" from the same weights. Double-pass boosts math. Triple-pass boosts emotional reasoning. Interleaved doubling (13,13,14,14,15,15,16) creates a pure math specialist. Same model, same VRAM, different routing. As far as I can see that's not implied by the original post. But that's beside the point: quoting the bit where the poster says "h…

You're right that my quote was misleading, I overlooked "the weird part" in the post because it didn't seem new to me either.

Here's the section in the original post that covers it: https://dnhkng.github.io/posts/rys/#the-brain-scanner All heatmaps are split by tasks and show an optimal point for each. The resulting routing he chose is a trade-off for both tasks, there isn't much else to do unless you intend to train a router anyway.

> So the ‘math organ’ has boundaries on both sides. Too few layers and you get nothing — you’ve cut into the circuit and it can’t complete its operation. Too many layers and you also get nothing — you’ve included tissue from a neighbouring circuit that doesn’t belong. Pre-training carved these structures out of the layer stack, and they only work whole. It also doesn’t translate to other tasks, as the heatmap for EQ scores doesn’t have this patch.

Re: Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

#74

Post by getnormality »

4bpp wrote: post #27

Assuming the benchmarks are sound (rather than capturing a fluke), the provided explanation still does not pass the smell test. As far as I can tell, there is nothing about the training process of these models that would encourage them to make the output of any layer apart from (n-1) meaningful as the input of layer n, unless perhaps these layers were initialised as identity and the training process did not get to ch…

> something happened to the model in training (RLHF?) that forcefully degraded its reasoning performance

I've been seeing more people speculating like this and I don't understand why. What evidence do we have for RLHF degrading performance on a key metric like reasoning? Why would this be tolerated by model developers?

Can someone point to an example of an AI researcher saying "oops, RLHF forcefully degrades reasoning capabilities, oh well, nothing we can do"?

It strikes me as conspiracist reasoning, like "there's a car that runs on water but they won't sell it because it would destroy oil profits".

Re: Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

#76

Post by WithinReason »

4bpp wrote: post #27

Assuming the benchmarks are sound (rather than capturing a fluke), the provided explanation still does not pass the smell test. As far as I can tell, there is nothing about the training process of these models that would encourage them to make the output of any layer apart from (n-1) meaningful as the input of layer n, unless perhaps these layers were initialised as identity and the training process did not get to ch…

Some transformers have a block recurrent structure, here is a paper that made a similar observation recently:

https://www.alphaxiv.org/abs/2512.19941

Re: Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

#77

Post by wavemode »

the_harpia_io wrote: post #75
samlinnfer wrote: post #69

Earlier quoted context omitted.

@dang ban this AI bot spam >is doing a lot of work here. tidy story, fits the result That's 100% AI talk

[flagged]

Can you cite which papers you're referring to?

Re: Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

#78

Post by WithinReason »

jstanley wrote: post #66
4bpp wrote: post #27

Assuming the benchmarks are sound (rather than capturing a fluke), the provided explanation still does not pass the smell test. As far as I can tell, there is nothing about the training process of these models that would encourage them to make the output of any layer apart from (n-1) meaningful as the input of layer n, unless perhaps these layers were initialised as identity and the training process did not get to ch…

> As far as I can tell, there is nothing about the training process of these models that would encourage them to make the output of any layer apart from (n-1) meaningful as the input of layer n Right, I had the same thought. Even if the output was in the same "format", does the LLM even have any way to know which order the outputs will go in? The ordering of the nodes is part of our representation of the network, it'…

These layers are residual layers, so what a layer does is:

x = x + layer(x)

so it's not too surprising that they can be used recurrently

Re: Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training

#80

Post by abhikul0 »

Karuma wrote: post #71
abhikul0 wrote: post #37

Earlier quoted context omitted.

I only know of this one: https://github.com/shootthesound/comfyUI-Realtime-Lora . Haven't played with any layer manipulation though.

I was thinking more like this one: https://github.com/AdamNizol/ComfyUI-Anima-Enhancer/ "It adds the Anima Layer Replay Patcher, which can enhance fine detail and coherence by replaying selected internal blocks during denoising."

I tried out the one I linked with sd1.5 today, moved the sliders around like a total noob and got pretty bad results but I found no way to "replay" any of the layers like the one you linked, so thanks for the link. Must take a lot of trial & errors haha. I'll check it out, assuming it works for the anima preview 2 too.
Post reply on HN