If I pay for the chain of thought, I want to see the chain of thought. Simple. How would I know if it happened at all? Trust OpenAI? LOL
Learning to Reason with LLMs
101–110 of 1001 posts
Re: Learning to Reason with LLMs
#102A lot of skepticism here, but these are astonishing results! People should realize we’re reaching the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark”. And as anyone who’s spent time using Claude 3.5 Sonnet / GPT-4o can attest, these things really are useful and smart! (And, if these results hold up, O1 is much, much smarter.) This is a nerve-wracking time to be a knowled…
Even without AI, it's gotten ~10,000 times easier to write software than in the 1950s (eg. imagine trying to write PyTorch code by hand in IBM 650 assembly), but the demand for software engineering has only increased, because demand increases even faster than supply does. Jevons paradox: https://en.wikipedia.org/wiki/Jevons_paradox
> it's gotten ~10,000 times easier to write software than in the 1950s
It seems many of the popular tools want to make writing software harder than in the 2010s, though. Perhaps their stewards believe that if they keep making things more and more unnecessarily complicated, LLMs won't be able to keep up?
Re: Learning to Reason with LLMs
#103> we are releasing an early version of this model, OpenAI o1-preview, for immediate use in ChatGPT Awesome!
Read "immediate" in "immediate use" in the same way as "open" in "OpenAI".
Re: Learning to Reason with LLMs
#104after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring, we have decided not to show the raw chains of thought to users.
We're not going to give you training data... for a better user experience.
Re: Learning to Reason with LLMs
#105A lot of skepticism here, but these are astonishing results! People should realize we’re reaching the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark”. And as anyone who’s spent time using Claude 3.5 Sonnet / GPT-4o can attest, these things really are useful and smart! (And, if these results hold up, O1 is much, much smarter.) This is a nerve-wracking time to be a knowled…
I have written a ton of evaluations and run countless benchmarks and I'm not even close to convinced that we're at
> the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark”
so much as we're over-fitting these bench marks (and in many cases fishing for a particular way of measuring the results that looks more impressive).
While it's great that the LLM community has so many benchmarks and cares about attempting to measure performance, these benchmarks are becoming an increasingly poor signal.
> This is a nerve-wracking time to be a knowledge worker for sure.
It might because I'm in this space, but I personally feel like this is the best time to working in tech. LLMs still are awful at things requiring true expertise while increasingly replacing the need for mediocre programmers and dilettantes. I'm increasingly seeing the quality of the technical people I'm working with going up. After years of being stuck in rooms with leetcode grinding TC chasers, it's very refreshing.
Re: Learning to Reason with LLMs
#106A lot of skepticism here, but these are astonishing results! People should realize we’re reaching the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark”. And as anyone who’s spent time using Claude 3.5 Sonnet / GPT-4o can attest, these things really are useful and smart! (And, if these results hold up, O1 is much, much smarter.) This is a nerve-wracking time to be a knowled…
> People should realize we’re reaching the point where LLMs are surpassing humans in any task limited in scope enough to be a “benchmark
This seems like a bold statement considering we have so few benchmarks, and so many of them are poorly put together.
Re: Learning to Reason with LLMs
#107I am not up-to-speed on CoT side but is this similar to how perplexity does it ie.
- generate a plan - execute the steps in plan (search internet, program this part, see if it is compilable)
each step is a separate gpt inference with added context from previous steps.
is O1 same? or does it do all this in a single inference run?
Re: Learning to Reason with LLMs
#108> we are releasing an early version of this model, OpenAI o1-preview, for immediate use in ChatGPT Awesome!
Read "immediate" in "immediate use" in the same way as "open" in "OpenAI".
You can use it, I just tried a few minutes ago. It's apparently limited to 30 messages/week, though.
Re: Learning to Reason with LLMs
#109Finally, a Claude competitor!
Re: Learning to Reason with LLMs
#110Are we ready yet to admit Turing test has been passed?
The Turing Test (which involves fooling a human into thinking they are talking to another human rather than a computer) has been routinely passed by very rudimentary "AI" since as early as 1991. It has no relevance today.