as more llms can, thats more everyone tryes to measure, so anyoone want to measure why they are better then others. Marketing stuff etc etc
The Benchmarkpocalypse
61–70 of 74 posts
Re: The Benchmarkpocalypse
#62Earlier quoted context omitted.
Ungrounded LLM outputs are a bit like your dreams. Without anything to test hypotheses against, stuff can pop in and out of existence and physics is just advice. Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own. Take yourself out of that loop so you only find out once it's sure.
That’s exactly why I don’t believe LLMs will cure cancer anytime soon, make terrible lawyers, shouldn’t be trusted for medical decisions, etc. software and maths are some really the niches where we have great, battle tested, reliable validation tools. That’s not the case for “softer” domains
Software is a very "spiky domain"; things either work or fail, and there is sharp delineation between and easy verification.
Hm. Two orthogonal properties! This sounds like a 2x2 matrix!
Let's swap hard/easy around & explore the 4 possibilities...
There are domains with sharp delineation and hard verification; they are not at risk until AI gets much better. Humans operate in these domains by applying tremendous deep thought and subjective judgment - our superpower.
Domains with soft delineation and easy verification are most at risk: "it's a picture of a cat" remains true through a wide range of perturbations - eg. skewing the image or moving it across a pixel or correcting its white balance or even changing the cat. AI music? Lots of domains already solved by AI here but they're also not that meaty.
My prediction is the next interesting stuff will happen where verification is hard but there's no sharp delineation. It's the world of "I'll know it when I see it". Good customer service?
Re: The Benchmarkpocalypse
#63Earlier quoted context omitted.
This is a "don't make me tap the sign" moment. LLMs are next token prediction models. If there are factual errors, confused ideas, etc. in the preceding tokens, that will affect the generation of subsequent tokens, and the error accumulates. Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a…
I don't see how "it's just token prediction" is relevant to that at all. Sure, the model can go wrong, but sometimes it's able to realise that and correct its course. Stronger models are better at doing this. People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley? To say the LLM has immutable limits because it only predicts the next token and can't backtrack is like saying…
> People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley?
Thus is why I think calling out that it is just predicting tokens is relevant. LLM are trained on human written communication. Humans say they found the root cause often, so the LLMs say it often. Even when not true.
Idk if thats true or not, but LLMs mimicking some average speech written pattern isnt crazy to me.
Re: The Benchmarkpocalypse
#64Earlier quoted context omitted.
That’s exactly why I don’t believe LLMs will cure cancer anytime soon, make terrible lawyers, shouldn’t be trusted for medical decisions, etc. software and maths are some really the niches where we have great, battle tested, reliable validation tools. That’s not the case for “softer” domains
Software is a very "spiky domain"; things either work or fail, and there is sharp delineation between and easy verification. Hm. Two orthogonal properties! This sounds like a 2x2 matrix! Let's swap hard/easy around & explore the 4 possibilities... There are domains with sharp delineation and hard verification; they are not at risk until AI gets much better. Humans operate in these domains by applying tremendous deep…
Eh, that’s a very interesting way to differentiate, I will steal your explanation next time I have that discussion, if you don’t mind!
Re: The Benchmarkpocalypse
#65Can anyone explain (or link to an article which explains) why telling Claude things like not to cheat or that there is a holdout set makes the output better? People say that an LLM is just a statistical next token predictor. I've read the article by Stephen Wolfram [0] on this, which makes sense to me, but then I can't see how 'more complex' behavior like cheating or not cheating can arise. [0] : https://writings.ste…
AI is trained off a massive corpus of internet exchanges. If you tell people youre going to thoroughly inspect their work or are able to check if theyre right, only people with high confidence will respond. I imagine exchanges where the OP is more involved in fact checking have a higher proportion of thoughtful and honest answers.
Much of the ‘secret sauce’ at the AI labs is not from the corpus on which they are trained but from the reinforcement training done afterwards. This step can influence the ‘personalities’ of the models and it is what makes them better at being ‘agents’, able to string together various individual steps to achieve your goal.
You could imagine that telling the LLM you have a holdout test makes it ‘feel’ more like an environment in which it was being RLed and therefore makes it better at seeking the reward by doing a good job.
Re: The Benchmarkpocalypse
#66> it's become easier than ever to make serious performance gains Is that true and if yes why? I was under the impression that it would become more difficult over time to make serious performance gains, which would also fit with reaching for benchmark hacking rather than relying on natural gains.
For a program that has not received much optimisation effort, the thing limiting performance gains is developer time and skill. It is not uncommon to have some small bit of code that could matter a lot, eg maybe you are doing something like escaping a string in a special way or processing lots of json. A human might write a simple character-at-a-time loop for the first case and might use some DOM-based api for the second case. What would have once required quite skilful work to optimise can now be done by siccing an LLM on the problem, where it can achieve a reasonable result (eg SIMD+expand-bits in the first case, and a good streaming api in the second) so long as you have good tests (fuzzing/property tests, production examples) and an ok benchmark (ideally a representative sample of production data plus some edge cases) you should get small results. The failure mode for optimisation described in this post is for harder optimisation problems with competing benchmarks.
LLMs are also pretty willing to do drudgery and are quite good at using perf, looking at traces, etc. they are often pretty stupid but if you throw a lot of stupid effort at a problem you can still achieve better results than the status quo of throwing ~zero smart effort at it.
Re: The Benchmarkpocalypse
#67I had a similar experience in search and found even holdouts can be overfit to. IE through brute force, it may not see the holdout, but if you gate a change on holdout acceptance it will land on a solution that’s overfit to it by somewhat random chance. The other problem is that holdouts / data inaccessible to the agent isn’t easy to do in most coding agents. It’s not as simple as splitting training data 80% and givi…
article description is low on details but i don't think what you and OP are describing can be categorized as overfitting in the statistical sense and more like "reward hacking" by the model. like you said more likely than not the agent had access to bench source code and just fitted solution to that.
Re: The Benchmarkpocalypse
#68I stopped trusting benchmarks ever since LLMs started speaking alien like English, if I can't understand what they are saying how can I trust them.
I wouldn't generalize this to all LLMs. So far I only saw Anthropic ones affected.
Re: The Benchmarkpocalypse
#69Earlier quoted context omitted.
I don't see how "it's just token prediction" is relevant to that at all. Sure, the model can go wrong, but sometimes it's able to realise that and correct its course. Stronger models are better at doing this. People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley? To say the LLM has immutable limits because it only predicts the next token and can't backtrack is like saying…
> People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley? Thus is why I think calling out that it is just predicting tokens is relevant. LLM are trained on human written communication. Humans say they found the root cause often, so the LLMs say it often. Even when not true. Idk if thats true or not, but LLMs mimicking some average speech written pattern isnt crazy to me.
LLMs aren’t just blindly parroting human speech, though -- they’re not trivial Markov chains any more, copying words and phrases at the surface level.
You can argue back and forth about whether LLMs are actually “conscious”, whatever that means, but it’s clear that they can be tremendously effective and useful. They can use language to get stuff done.
Re: The Benchmarkpocalypse
#70Earlier quoted context omitted.
Ungrounded LLM outputs are a bit like your dreams. Without anything to test hypotheses against, stuff can pop in and out of existence and physics is just advice. Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own. Take yourself out of that loop so you only find out once it's sure.
That’s exactly why I don’t believe LLMs will cure cancer anytime soon, make terrible lawyers, shouldn’t be trusted for medical decisions, etc. software and maths are some really the niches where we have great, battle tested, reliable validation tools. That’s not the case for “softer” domains
The only people who think LLMs would make good lawyers are the people selling LLMs. The more practical among us recognize that LLMs are our amazing tools for searching through and making sense of large amounts of text with a high level of sophistication, which can significantly enhance the productivity of a human lawyer.