41–50 of 137 posts
Anything you can verify to be right or wrong can be done by a model. All benchmarks will be saturated
Examples:
- predict a coinflip: easy to verify, hard to learn
- earn $100: easy to verify, hard to learn
- increase paid subscriptions in an A/B test: easy to verify, hard to learn
I won't get into it, but there are many properties beyond verifiability that are needed to saturate a benchmark.
Is solving a snake like puzzle game in the least number of moves really what defines intelligence?
It's pretty close to how we measure IQ. The standard test is basically a series of spatial puzzles. I know there's a lot of people who complain that we're moving goalposts, but I think that the progress in LLMs really just shows that we don't know how to really measure intelligence in the first place, if we understand it as "human-like agency / ingenuity / adaptability". For decades, we saw the Turing test as the pro…
It was a useful lesson that whatever IQ tests measure, it is completely devoid of value or interest to me.
none 35.2%, $49,791 96.7%, $23,457
Earlier quoted context omitted.
DeepSeek v4 Flash recently had a similar "more reasoning is cheaper" curve. It's a fun counterintuition.
What is the intuition. Higher quality turns due to more reasoning results in significantly fewer turns taken?
Anything you can verify to be right or wrong can be done by a model. All benchmarks will be saturated
Disagree. Examples: - predict a coinflip: easy to verify, hard to learn - earn $100: easy to verify, hard to learn - increase paid subscriptions in an A/B test: easy to verify, hard to learn I won't get into it, but there are many properties beyond verifiability that are needed to saturate a benchmark.
- earn $100: easy to verify, hard to learn
- increase paid subscriptions in an A/B test: easy to verify, hard to learn
but we both know these examples go against the spirit of my point
Earlier quoted context omitted.
Then why is unemployment around 4%? You believe we have AGI and yet it can’t do anyone’s job?
Didn’t I just see a thing about how actual unemployment is at like 24% a few days ago?
(I have not gone down the rabbit hole to understand how they achieve that 24% number)
99.9% with the right harness? Ok, we're at AGI then. Prediction: We will now see the goalposts moved towards "well, a human costs less / is more efficient" - that will prevail for a few months until they come up with some other test that humans can do easily but is hard for the bots. This cycle will continue for ever and in 25 years, despite having hyper intelligent embodied robots or whatever, we'll still be arguing…
TLDR: The official ARC harness throws away old context and reasoning. No real-world harness is this bad, the model has to re-learn the game repeatedly. OpenAI basically just added standard compaction. Their harness is still "general".
The instant/no reasoning performed extremely well none 35.2%, $49,791 96.7%, $23,457 35.2% on the standard harness, that's above Opus 5 on high.
Earlier quoted context omitted.
Disagree. Examples: - predict a coinflip: easy to verify, hard to learn - earn $100: easy to verify, hard to learn - increase paid subscriptions in an A/B test: easy to verify, hard to learn I won't get into it, but there are many properties beyond verifiability that are needed to saturate a benchmark.
these just need more compute: - earn $100: easy to verify, hard to learn - increase paid subscriptions in an A/B test: easy to verify, hard to learn but we both know these examples go against the spirit of my point