Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

dnhkng.github.io

Post reply on HN
Print view View original

71–80 of 137 posts

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#71

Post by dnhkng »

mysteria wrote: post #61

The astounding thing about Goliath wasn’t that is was a huge leap in performance, it was that the damn thing functioned at all. To this day, I still don’t understand why this didn’t raise more eyebrows. This wasn't something I really dug into in great detail but I remember my surprise back then at how all those merged models and those "expanded" models like Goliath still generated coherent output. IMO those were more…

It's still non-trivial, as multi-digit numbers can be constructed a huge combination of valid tokens.

The code in the blog helps derive useful metrics from partial answers.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#72

Post by dnhkng »

iamjackg wrote: post #60

I find the concept of LLM "brain surgery" fascinating, precisely because of how opaque the network is. One of the first things I did back when llama.cpp first got vision model support was hack the code to zero out (or otherwise modify) random numbers in the image embedding generated by the projector and then ask the LLM to describe the image. It was absolutely fascinating . It would go from a normal description of th…

Yes, it's an amazing time to be a hacker!

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#73

Post by gavinray »

rapatel0 wrote: post #66
dnhkng wrote: post #27

Earlier quoted context omitted.

Yes, I've tried duplicating indvidual layers, but its not useful. I think this hasn't been tried before because it's totally unintuitive that feeding the output from later layers into previous ones would actually do anything. And in fact, it usually is detrimental. I guess it takes really bored hobbyists with too much compute to check this stuff. I have done some interesting work on applying multiple layer duplicatio…

Clarification. Duplicating multiple groups of layers in a "reasoning" loop Normal: L1 -> L2 -> L3 -> L4 -> out Unrolled (current framing): L1 -> [L2->L3] -> [L2->L3] -> L4 -> out Looped (proposed): -- [L2->L3] x N --> L4 -> out "reasoning loop" Note: ascii rendering HN is not trivial

The commenter "Skerit" below linked to a recent implementation of this:

https://ouro-llm.github.io/

See the left-hand side of the diagram here, which is your exact proposal:

https://ouro-llm.github.io/static/images/ouro_main.png

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#74

Post by imranq »

Amazing write up and i wish more people showed the process for discovery which is often even more interesting than the result itself

Still the result is really interesting being able to stack abstract reasoning and get better performance and the heat maps to show the prob results

The academic literature seems to be catching up:

- *[SOLAR / DUS (Kim et al., 2023)](https://arxiv.org/abs/2312.15166)* — duplicated transformer layers to build a 10.7B model that outperformed 30B parameter baselines.

- *[The Curse of Depth (2025)](https://arxiv.org/abs/2502.05795)* — explains why this works: Pre-LN causes deep transformer layers to converge toward identity functions, meaning middle layers are where real computation happens, and duplicating them concentrates that capacity.

- *[Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (Geiping et al., NeurIPS 2025)](https://arxiv.org/abs/2502.05171)* — takes the idea to its logical conclusion: a model trained with a single recurrent block repeated at inference time, scaling reasoning depth without adding parameters.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#75

Post by janalsncm »

Lerc wrote: post #51

I have had broadly the same intuitions on the use of middle layers, but haven't had much luck with the tiny models that I can run on my hardware. There's a video on YouTube https://www.youtube.com/watch?v=pDsTcrRVNc0 about a looping layer models, after watching that I poured some thoughts off the top of my head into a comment which, of course, promptly sunk without a trace. I'll repost the gist of them here. If you g…

That is an interesting idea. I suspect if we relax the constraint that most of the layers in a loop will be in order, there is a combinatorial explosion issue.

But we could still try it out: randomize the order we call the transformer blocks, and see if it affects performance. If not, that’s extremely interesting.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#76

Post by janalsncm »

mysteria wrote: post #61

The astounding thing about Goliath wasn’t that is was a huge leap in performance, it was that the damn thing functioned at all. To this day, I still don’t understand why this didn’t raise more eyebrows. This wasn't something I really dug into in great detail but I remember my surprise back then at how all those merged models and those "expanded" models like Goliath still generated coherent output. IMO those were more…

It’s a good spot for hobbyists to fill in the gaps. Maybe it’s not interesting enough for academics to study, and for corporate ML they would probably just fine tune something that exists rather than spending time on surgery. Even Chinese labs that are more resource constrained don’t care as much about 4090-scale models.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#77

Post by [deleted] »

dnhkng wrote: post #46
dinobones wrote: post #41

Earlier quoted context omitted.

why tho? it's just an alternate alphabet/set of symbols.

Because its generally expected that models only work 'in distribution', i.e. they work on stuff they have previously seen. They almost certainly have never seen regular conversations in Base64 in their training set, so its weird that it 'just works'. Does that make sense?

[deleted]

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#78

Post by janalsncm »

It would be extremely interesting if we could use this kind of model surgery approach to tack on additional modalities. For example, adding vision to a text only model.

Another very interesting thing would be modulating compute at the token level. Default is 0 loops, maybe 1 loop is better, and 10 loops is even better than that.

Re: Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs

#80

Post by fweimer »

dnhkng wrote: post #46
dinobones wrote: post #41

Earlier quoted context omitted.

why tho? it's just an alternate alphabet/set of symbols.

Because its generally expected that models only work 'in distribution', i.e. they work on stuff they have previously seen. They almost certainly have never seen regular conversations in Base64 in their training set, so its weird that it 'just works'. Does that make sense?

If you do not properly MIME-decode email, you end up with at least some base64-encoded conversations.
Post reply on HN