[deleted]
Learning to Reason with LLMs
51–60 of 1001 posts
Re: Learning to Reason with LLMs
#52after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring, we have decided not to show the raw chains of thought to users.
"Open"AI is such a comically ironic name at this point.
Re: Learning to Reason with LLMs
#53A lot of skepticism here, but these are astonishing results! People should realize we’re reaching the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark”. And as anyone who’s spent time using Claude 3.5 Sonnet / GPT-4o can attest, these things really are useful and smart! (And, if these results hold up, O1 is much, much smarter.) This is a nerve-wracking time to be a knowled…
I cannot, in fact, attest that they are useful and smart. LLMs remain a fun toy for me, not something that actually produces useful results.
I have been deploying useful code from LLMs right and left over the last several months. They are a significant force accelerator for programmers if you know how to prompt them well.
Re: Learning to Reason with LLMs
#54Sounds great, but so does their "new flagship model that can reason across audio, vision, and text in real time" announced in May. [0]
Re: Learning to Reason with LLMs
#55Wouldn't this introduce new economics into the LLM market?
I.e. if the "thinking loop" budget is parameterized, users might pay more (much more) to spend more compute on a particular question/prompt.
Re: Learning to Reason with LLMs
#56yeah this is kinda cool i guess but 808 elo is still pretty bad for a model that can supposedly code like a human, i mean 11th percentile is like barely scraping by, and what even is the point of simulating codeforces if youre just gonna make a model that can barely compete with a decent amateur, and btw what kind of contest allows 10 submissions, thats not how codeforces works, and what about the time limits and mem…
808 ELO was for GPT-4o.
I would suggest re-reading more carefully
Re: Learning to Reason with LLMs
#57A lot of skepticism here, but these are astonishing results! People should realize we’re reaching the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark”. And as anyone who’s spent time using Claude 3.5 Sonnet / GPT-4o can attest, these things really are useful and smart! (And, if these results hold up, O1 is much, much smarter.) This is a nerve-wracking time to be a knowled…
Is it? They talk about 10k attempts to reach gold medal status in the mathematics olympiad, but zero shot performance doesn't even place it in the upper 50th percentile.
Maybe I'm confused but 10k attempts on the same problem set would make anyone an expert in that topic? It's also weird that zero shot performance is so bad, but over a lot of attempts it seems to get correct answers? Or is it learning from previous attempts? No info given.
Re: Learning to Reason with LLMs
#58Do people see the new models in the web interface? Mine still shows the old models (I'm a paid subscriber).
Re: Learning to Reason with LLMs
#59Won't be surprised to see all these hand-picked results and extreme expectations to collapse under scenarios involving highly safety critical and complex demanding tasks requiring a definite focus on detail with lots of awareness, which what they haven't shown yet.
So let's not jump straight into conclusions with these hand-picked scenarios marketed to us and be very skeptical.
Not quite there yet with being able to replace truck drivers and pilots for self-autonomous navigation in transportation, aerospace or even mechanical engineering tasks, but it certainly has the capability in replacing both typical junior and senior software engineers in a world considering to do more with less software engineers needed.
But yet, the race to zero will surely bankrupt millions of startups along the way. Even if the monthly cost of this AI can easily be as much as a Bloomberg terminal to offset the hundreds of billions of dollars thrown into training it and costing the entire earth.
Re: Learning to Reason with LLMs
#60Generating more "think out loud" tokens and hiding them from the user... Idk if I'm "feeling the AGI" if I'm being honest. Also... telling that they choose to benchmark against CodeForces rather than SWE-bench.
Why not? Isn't that basically what humans do? Sit there and think for a while before answering, going down different branches/chains of thought?
This new approach is showing:
1) The "bitter lesson" may not be true, and there is a fundamental limit to transformer intelligence.
2) The "bitter lesson" is true, and there just isn't enough data/compute/energy to train AGI.
All the cognition should be happening inside the transformer. Attention is all you need. The possible cognition and reasoning occurring "inside" in high dimensions is much more advanced than any possible cognition that you output into text tokens.
This feels like a sidequest/hack on what was otherwise a promising path to AGI.