This is a prompt engineering saas
Learning to Reason with LLMs
141–150 of 1001 posts
Re: Learning to Reason with LLMs
#142A lot of skepticism here, but these are astonishing results! People should realize we’re reaching the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark”. And as anyone who’s spent time using Claude 3.5 Sonnet / GPT-4o can attest, these things really are useful and smart! (And, if these results hold up, O1 is much, much smarter.) This is a nerve-wracking time to be a knowled…
Is it? They talk about 10k attempts to reach gold medal status in the mathematics olympiad, but zero shot performance doesn't even place it in the upper 50th percentile. Maybe I'm confused but 10k attempts on the same problem set would make anyone an expert in that topic? It's also weird that zero shot performance is so bad, but over a lot of attempts it seems to get correct answers? Or is it learning from previous a…
The blog says "With a relaxed submission constraint, we found that model performance improved significantly. When allowed 10,000 submissions per problem, the model achieved a score of 362.14 – above the gold medal threshold – even without any test-time selection strategy."
I am interpreting this to mean that the model tried 10K approaches to solve the problem, and finally selected the one that did the trick. Am I wrong?
Re: Learning to Reason with LLMs
#143Do people see the new models in the web interface? Mine still shows the old models (I'm a paid subscriber).
Not yet, it's still not available in the web interface. I think they're rolling it out step by step.
Anyway, the usage limits are pretty ridiculous right now, which makes it even more frustrating.
Re: Learning to Reason with LLMs
#1442018 - gpt1 2019 - gpt2 2020 - gpt3 2022 - gpt3.5 2023 - gpt4 2023 - gpt4-turbo 2024 - gpt-4o 2024 - o1 Did OpenAI hire Google's product marketing team in recent years?
Makes sense to me actually. This is a different product. It doesn't respond instantly.
It fundamentally makes sense to separate these two products in the AI space. There will obviously be a speed vs quality trade-off with a variety of products across the spectrum over time. LLMs respond way too fast to actually be expected to produce the maximum possible quality of a response to complex queries.
Re: Learning to Reason with LLMs
#145after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring, we have decided not to show the raw chains of thought to users.
Lame but not atypical of OpenAI. Too bad, but I'm expecting competitors to follow with this sort of implementation and better. Being able to view the "reasoning" process and especially being able to modify it and re-render the answer may be faster than editing your prompt a few times until you get the desired response, if you even manage to do that.
Re: Learning to Reason with LLMs
#146https://openai.com/index/introducing-openai-o1-preview/ > ChatGPT Plus and Team users will be able to access o1 models in ChatGPT starting today. Both o1-preview and o1-mini can be selected manually in the model picker, and at launch, weekly rate limits will be 30 messages for o1-preview and 50 for o1-mini. We are working to increase those rates and enable ChatGPT to automatically choose the right model for a given p…
It's probably running several lines of COT. I imagine, each single message you send is probably at __least__ 10x to the actual model. So in reality it's like 300 messages, and honestly it's probably 100x, given how constrained they're being with usage.
Re: Learning to Reason with LLMs
#147> Unless otherwise specified, we evaluated o1 on the maximal test-time compute setting. Maximal test time is the maximum amount of time spent doing the “Chain of Thought” “reasoning”. So that’s what these results are based on. The caveat is that in the graphs they show that for each increase in test-time performance, the (wall) time / compute goes up exponentially . So there is a potentially interesting play here. Th…
I interpreted it to suggest that the product might include a user-facing “maximum test time” knob.
Generating problem sets for kids? You might only need or want a basic level of introspection, even though you like the flavor of this model’s personality over that of its predecessors.
Problem worth thinking long, hard, and expensively about? Turn that knob up to 11, and you’ll get a better-quality answer with no human-in-the-loop coaching or trial-and-error involved. You’ll just get your answer in timeframes closer to human ones, consuming more (metered) tokens along the way.
Re: Learning to Reason with LLMs
#148A lot of skepticism here, but these are astonishing results! People should realize we’re reaching the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark”. And as anyone who’s spent time using Claude 3.5 Sonnet / GPT-4o can attest, these things really are useful and smart! (And, if these results hold up, O1 is much, much smarter.) This is a nerve-wracking time to be a knowled…
Is it? They talk about 10k attempts to reach gold medal status in the mathematics olympiad, but zero shot performance doesn't even place it in the upper 50th percentile. Maybe I'm confused but 10k attempts on the same problem set would make anyone an expert in that topic? It's also weird that zero shot performance is so bad, but over a lot of attempts it seems to get correct answers? Or is it learning from previous a…
Even if you disregard the Olympiad performance OpenAI-O1 is, if the charts are to be believed, a leap forward in intelligence. Also bear in mind that AI researchers are not out of ideas on how to make models better and improvements in AI chips are the metaphorical tide that lifts all boats. The trend is the biggest story here.
I get the AI skepticism because so much tech hype of recent years turned out to be hot air (if you're generous, obvious fraud if you're not). But AI tools available toady, once you get the hang of using them, are pretty damn amazing already. Many jobs can be fully automated with AI tools that exist today. No further breakthroughs required. And although I still don't believe software engineers will find themselves out of work anytime soon, I can no longer completely rule it out either.
Re: Learning to Reason with LLMs
#149A lot of skepticism here, but these are astonishing results! People should realize we’re reaching the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark”. And as anyone who’s spent time using Claude 3.5 Sonnet / GPT-4o can attest, these things really are useful and smart! (And, if these results hold up, O1 is much, much smarter.) This is a nerve-wracking time to be a knowled…
[deleted]
Re: Learning to Reason with LLMs
#150The model performance is driven by chain of thought, but they will not be providing chain of thought responses to the user for various reasons including competitive advantage. After the release of GPT4 it became very common to fine-tune non-OpenAI models on GPT4 output. I’d say OpenAI is rightly concerned that fine-tuning on chain of thought responses from this model would allow for quicker reproduction of their resu…
It'd be helpful if they exposed a summary of the chain-of-thought response instead. That way they'd not be leaking the actual tokens, but you'd still be able to understand the outline of the process. And, hopefully, understand where it went wrong.
They do, according to the example