This is the sort of reasoning needed to solve the ARC AGI benchmark.
Learning to Reason with LLMs
111–120 of 1001 posts
Re: Learning to Reason with LLMs
#112after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring, we have decided not to show the raw chains of thought to users.
This also makes them less useful because I can’t just click stop generation when they make a logical error re: coding.
You wouldn't do that to this model. It finds its own mistakes and corrects them as it is thinking through things.
Re: Learning to Reason with LLMs
#113A lot of skepticism here, but these are astonishing results! People should realize we’re reaching the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark”. And as anyone who’s spent time using Claude 3.5 Sonnet / GPT-4o can attest, these things really are useful and smart! (And, if these results hold up, O1 is much, much smarter.) This is a nerve-wracking time to be a knowled…
I like your phrasing - "any task limited in scope enough to be a 'benchmark'". Exactly! This is the real gap with LLMs, and will continue to be an issue with o1 -- sure, if you can write down all of the relevant context information you need to perform some computation, LLMs should be able to do it. In other words, LLMs are calculators!
I'm not especially nerve-wracked about being a knowledge worker, because my day-to-day doesn't consist of being handed a detailed specification of exactly what is required, and then me 'computing' it. Although this does sound a lot like what a product manager does!
Re: Learning to Reason with LLMs
#114Wait, are they comparing 4o without CoT and o1 with built-in CoT?
yeah was wondering what 4o with a CoT in the prompt would look like.
Re: Learning to Reason with LLMs
#115Wouldn't this introduce new economics into the LLM market? I.e. if the "thinking loop" budget is parameterized, users might pay more (much more) to spend more compute on a particular question/prompt.
Yes, and note the large price increase
Re: Learning to Reason with LLMs
#116A lot of skepticism here, but these are astonishing results! People should realize we’re reaching the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark”. And as anyone who’s spent time using Claude 3.5 Sonnet / GPT-4o can attest, these things really are useful and smart! (And, if these results hold up, O1 is much, much smarter.) This is a nerve-wracking time to be a knowled…
I cannot, in fact, attest that they are useful and smart. LLMs remain a fun toy for me, not something that actually produces useful results.
Familiarize yourself with a tool which does half the prompting for you, e.g. cursor is pretty good at prompting claude 3.5 and it really does make code edits 10x faster (I'm not even talking about the fancy stuff about generating apps in 5 mins - just plain old edits.)
Re: Learning to Reason with LLMs
#117Interesting sequence from the Cipher CoT:
Third pair: 'dn' to 'i'
'd'=4, 'n'=14
Sum:4+14=18
Average:18/2=9
9 corresponds to 'i'(9='i')
But 'i' is 9, so that seems off by 1.
So perhaps we need to think carefully about letters.
Wait, 18/2=9, 9 corresponds to 'I'
So this works.
-----
This looks like recovery from a hallucination. Is it realistic to expect CoT to be able to recover from hallucinations this quickly?
Re: Learning to Reason with LLMs
#118A lot of skepticism here, but these are astonishing results! People should realize we’re reaching the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark”. And as anyone who’s spent time using Claude 3.5 Sonnet / GPT-4o can attest, these things really are useful and smart! (And, if these results hold up, O1 is much, much smarter.) This is a nerve-wracking time to be a knowled…
> And as anyone who’s spent time using Claude 3.5 Sonnet / GPT-4o can attest, these things really are useful and smart! (And, if these results hold up, O1 is much, much smarter.) This is a nerve-wracking time to be a knowledge worker for sure.
If you have to keep checking the result of an LLM, you do not trust it enough to give you the correct answer.
Thus, having to 'prompt' hundreds of times for the answer you believe is correct over something that claims to be smart - which is why it can confidently convince others that its answer is correct (even when it can be totally erroneous).
I bet if Google DeepMind announced the exact same product, you would equally be as skeptical with its cherry-picked results.
Re: Learning to Reason with LLMs
#119> Unless otherwise specified, we evaluated o1 on the maximal test-time compute setting.
Maximal test time is the maximum amount of time spent doing the “Chain of Thought” “reasoning”. So that’s what these results are based on.
The caveat is that in the graphs they show that for each increase in test-time performance, the (wall) time / compute goes up exponentially.
So there is a potentially interesting play here. They can honestly boast these amazing results (it’s the same model after all) yet the actual product may have a lower order of magnitude of “test-time” and not be as good.
Re: Learning to Reason with LLMs
#120A near perfect on AMC 12, 1900 CodeForces ELO, and silver medal IOI competitor. In two years, we'll have models that could easily win IMO and IOI. This is __incredible__!!