Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Benchmarkpocalypse

danluu.com

Post reply on HN
Print view View original

31–40 of 74 posts

Re: The Benchmarkpocalypse

#31

Post by dgellow »

cadamsdotcom wrote: post #4
timfsu wrote: post #3

Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in…

Ungrounded LLM outputs are a bit like your dreams. Without anything to test hypotheses against, stuff can pop in and out of existence and physics is just advice. Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own. Take yourself out of that loop so you only find out once it's sure.

That’s exactly why I don’t believe LLMs will cure cancer anytime soon, make terrible lawyers, shouldn’t be trusted for medical decisions, etc. software and maths are some really the niches where we have great, battle tested, reliable validation tools. That’s not the case for “softer” domains

Re: The Benchmarkpocalypse

#32

Post by [deleted] »

paulddraper wrote: post #20
gwerbin wrote: post #19

Earlier quoted context omitted.

This is a "don't make me tap the sign" moment. LLMs are next token prediction models. If there are factual errors, confused ideas, etc. in the preceding tokens, that will affect the generation of subsequent tokens, and the error accumulates. Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a…

That’s a bit of a garbage in, garbage out moment.

[deleted]

Re: The Benchmarkpocalypse

#33

Post by dgellow »

puszczyk wrote: post #26
ouz-a wrote: post #16

I stopped trusting benchmarks ever since LLMs started speaking alien like English, if I can't understand what they are saying how can I trust them.

This article is about using LLMs to overfit for a specific benchmark (or make a custom software for niche use cases) though. Not about LLMs benchmaxxxing

Isn’t that the same? It’s a sort of recursive version of overfitting specific benchmarks

Re: The Benchmarkpocalypse

#34

Post by friendzis »

timfsu wrote: post #3

Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in…

> It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards

LLMs are trained to produce some form of output that is NOT nothing/dunno. The so called hyperspace collapses onto something, instead of collapsing onto lack of answer tokens.

Re: The Benchmarkpocalypse

#35

Post by iainmerrick »

gwerbin wrote: post #19
timfsu wrote: post #3

Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in…

This is a "don't make me tap the sign" moment. LLMs are next token prediction models. If there are factual errors, confused ideas, etc. in the preceding tokens, that will affect the generation of subsequent tokens, and the error accumulates. Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a…

I don't see how "it's just token prediction" is relevant to that at all.

Sure, the model can go wrong, but sometimes it's able to realise that and correct its course. Stronger models are better at doing this.

People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley?

To say the LLM has immutable limits because it only predicts the next token and can't backtrack is like saying we have immutable limits because we can't travel backwards in time. It's a true statement but not particularly relevant or helpful.

Re: The Benchmarkpocalypse

#36

Post by iainmerrick »

akoboldfrying wrote: post #8

I think an interesting direction for benchmarking is to take inspiration from metamorphic testing. Metamorphic testing is a way of extending property-based testing (in which you ask the test framework itself to automatically generate many random (input, expected output) pairs to test for you, instead of manually writing individual tests yourself) to handle situations where (a) it's hard to independently come up with…

I had a similar thought -- rather than fixed benchmarks, you want dynamically-generated tests, specifically designed to exercise newly-exposed corner cases. So the way forward might be antagonistic benchmarks generated by another LLM.

Re: The Benchmarkpocalypse

#37

Post by thatjoeoverthr »

timfsu wrote: post #3

Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in…

A lot of weirdness in their style is the fact they think in-band.

Notice it "calls its attacks" when writing, like Goku shouting "solar flare" and doing a solar flare.

To write an honest answer, it says, "Here's the honest answer", and proceeds from there.

If it plans to write a "kicker", first it says, "And here's the kicker", and then it writes the "kicker".

This gives an affect distinctly different from any professional writer because editors are specifically trained to _cut out_ explicated subtext.

You would expect "thinking" models and the agent harness (which conceals "thinking sections") would mitigate this but it's a deep seated behaviour.

Re: The Benchmarkpocalypse

#38

Post by frumiousirc »

stephantul wrote: post #5

Unfortunately, even a holdout set doesn’t protect you from overfitting, it just takes longer. Of course having a holdout set is better than not having one. It’s just not a silver bullet.

That is not what I read from danluu's words. He merely stated in the prompt that there is a holdout set and did not iterate to minimize error against the holdout set. In a prior attempt he prompted with only "don't overfit" to ill effect on the holdout eval. Did I misread?

Re: The Benchmarkpocalypse

#39

Post by kasey_junk »

iainmerrick wrote: post #35
gwerbin wrote: post #19

Earlier quoted context omitted.

This is a "don't make me tap the sign" moment. LLMs are next token prediction models. If there are factual errors, confused ideas, etc. in the preceding tokens, that will affect the generation of subsequent tokens, and the error accumulates. Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a…

I don't see how "it's just token prediction" is relevant to that at all. Sure, the model can go wrong, but sometimes it's able to realise that and correct its course. Stronger models are better at doing this. People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley? To say the LLM has immutable limits because it only predicts the next token and can't backtrack is like saying…

I have found it extremely helpful to keep the “token prediction machine” metaphor front and center in my work with llms, not because it expresses some fundamental limit of the technology but because it allows me to have a mental model about where and how to use the technology in my process.

Contrast this to the “chat” mental model. If you are chatting with someone and they lie to you, that is a transgression. A bad prediction is just a normal outcome. These are _human_ judgements but you address transgressions differently than normal but incorrect outcomes. That’s a helpful distinction for me when building my tooling.

If anything, that we are getting so much traction out of text prediction is the most magical part of llms. But it doesn’t change that you shouldn’t treat what they output the same as what a human says.

Post reply on HN