Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Benchmarkpocalypse

danluu.com

Post reply on HN
Print view View original

41–50 of 74 posts

Re: The Benchmarkpocalypse

#41

Post by iainmerrick »

kasey_junk wrote: post #39
iainmerrick wrote: post #35

Earlier quoted context omitted.

I don't see how "it's just token prediction" is relevant to that at all. Sure, the model can go wrong, but sometimes it's able to realise that and correct its course. Stronger models are better at doing this. People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley? To say the LLM has immutable limits because it only predicts the next token and can't backtrack is like saying…

I have found it extremely helpful to keep the “token prediction machine” metaphor front and center in my work with llms, not because it expresses some fundamental limit of the technology but because it allows me to have a mental model about where and how to use the technology in my process. Contrast this to the “chat” mental model. If you are chatting with someone and they lie to you, that is a transgression. A bad p…

Hmm, what specific differences does it make in practice?

For me, the huge one is that LLMs are currently bad at learning from experience. I don’t trust any kind of automatic MEMORY.md or whatnot; in fact I greatly prefer starting from a clean slate each time because the LLM’s baseline general knowledge is so good.

In terms of accuracy and “lying”, I don’t really see a huge difference. Most LLMs are unfortunately a bit sycophantic and over-confident, but you sometimes see that in people as well.

Re: The Benchmarkpocalypse

#42

Post by softwaredoug »

I had a similar experience in search and found even holdouts can be overfit to. IE through brute force, it may not see the holdout, but if you gate a change on holdout acceptance it will land on a solution that’s overfit to it by somewhat random chance.

The other problem is that holdouts / data inaccessible to the agent isn’t easy to do in most coding agents. It’s not as simple as splitting training data 80% and giving some to the agent and hiding 20%. The agent can figure out where its data came from and find ways to reconstruct / cheat the holdout data.

All the ways of doing this seem annoying: ie having a second project that accepts / rejects changes.

I opted to just build my own harness for these things to avoid overfitting.

https://softwaredoug.com/blog/2026/05/17/autoresearching-a-b...

Re: The Benchmarkpocalypse

#43

Post by kasey_junk »

iainmerrick wrote: post #41
kasey_junk wrote: post #39

Earlier quoted context omitted.

I have found it extremely helpful to keep the “token prediction machine” metaphor front and center in my work with llms, not because it expresses some fundamental limit of the technology but because it allows me to have a mental model about where and how to use the technology in my process. Contrast this to the “chat” mental model. If you are chatting with someone and they lie to you, that is a transgression. A bad p…

Hmm, what specific differences does it make in practice? For me, the huge one is that LLMs are currently bad at learning from experience. I don’t trust any kind of automatic MEMORY.md or whatnot; in fact I greatly prefer starting from a clean slate each time because the LLM’s baseline general knowledge is so good. In terms of accuracy and “lying”, I don’t really see a huge difference. Most LLMs are unfortunately a bi…

The biggest difference is I view any interaction with an llm that doesn’t produce an artifact that can be verified with skepticism.

That is I very rarely even use a “chat” interface anymore. Under the covers I’m still using the mainline agents but I’m treating them more like black box Unix tools that take text in and spit text out.

My interactions are evolving to have very small llm cores wrapped with old fashioned deterministic programming.

This makes the boundary/verification step more deterministic and less susceptible to bad predictions. But it also has the side effect that I rarely interact with llm output that has “tone” (for instance sycophantic output). That usually only happens when I’m debugging what went wrong, particularly around model upgrade cycles.

Re: The Benchmarkpocalypse

#44

Post by __alexs »

gwerbin wrote: post #19
timfsu wrote: post #3

Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in…

This is a "don't make me tap the sign" moment. LLMs are next token prediction models. If there are factual errors, confused ideas, etc. in the preceding tokens, that will affect the generation of subsequent tokens, and the error accumulates. Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a…

This line of thinking is quite confusing to me. If you keep following it then ultimately we must reckon with the reality that we only experience time in one direction. While it might be technically true that "there is only so much you can do with that" I think it might also be a completely useless statement to make.

Re: The Benchmarkpocalypse

#46

Post by ee334y5rthsrth »

timfsu wrote: post #3

Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in…

decrease temperature options to 0.0

Re: The Benchmarkpocalypse

#47

Post by fizzbuzzbarbazz »

__alexs wrote: post #44
gwerbin wrote: post #19

Earlier quoted context omitted.

This is a "don't make me tap the sign" moment. LLMs are next token prediction models. If there are factual errors, confused ideas, etc. in the preceding tokens, that will affect the generation of subsequent tokens, and the error accumulates. Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a…

This line of thinking is quite confusing to me. If you keep following it then ultimately we must reckon with the reality that we only experience time in one direction. While it might be technically true that "there is only so much you can do with that" I think it might also be a completely useless statement to make.

I think it provides a necessary pointer to the idea that we will need more than prediction to make a viable general AI - that the prediction model is useful, but has fundamental limitations.

Re: The Benchmarkpocalypse

#48

Post by internet_points »

softwaredoug wrote: post #42

I had a similar experience in search and found even holdouts can be overfit to. IE through brute force, it may not see the holdout, but if you gate a change on holdout acceptance it will land on a solution that’s overfit to it by somewhat random chance. The other problem is that holdouts / data inaccessible to the agent isn’t easy to do in most coding agents. It’s not as simple as splitting training data 80% and givi…

If you repeatedly use the same holdout and trigger acceptance on the holdout, it is no longer a holdout but just another training set.

Re: The Benchmarkpocalypse

#49

Post by softwaredoug »

internet_points wrote: post #48
softwaredoug wrote: post #42

I had a similar experience in search and found even holdouts can be overfit to. IE through brute force, it may not see the holdout, but if you gate a change on holdout acceptance it will land on a solution that’s overfit to it by somewhat random chance. The other problem is that holdouts / data inaccessible to the agent isn’t easy to do in most coding agents. It’s not as simple as splitting training data 80% and givi…

If you repeatedly use the same holdout and trigger acceptance on the holdout, it is no longer a holdout but just another training set.

Exactly

Re: The Benchmarkpocalypse

#50

Post by alexpotato »

> Even though the overall FRE regex engine has worse performance than the Rust regex crate, the gains you can get for specializing to your workload or use case mean that, in some cases, it could be reasonable to insert your own specialized regex engine somewhere, and the same goes for various other kinds of low-level software.

The above opinion matches what I've been thinking after months of using LLMs in production environments.

Instead of always defaulting to using the frontier model, you can either:

- have the LLM write a script/tool that covers 95% of the cases you expect to hit

- for the other 5%, fine tune a small local model for that remaining 5%

This has the benefits of:

1. lower token count over time

2. it's easy to see what the tool is actually doing b/c it's written in code

3. that code can be version controlled

4. you can gradually shift the fine tuned model workload to the code as it improves over time

Really, this is just what the "Manual Work is a Bug" [0] blog post described years ago but replacing "people doing the work and then automate via scripts" with "LLMs do the work -> automate".

0 - https://queue.acm.org/detail.cfm?id=3197520

Post reply on HN