I tested various Math Olympiad questions with Claude sonnet 3.5 and they all arrived at the correct solution. o1's solution was a bit better formulated, in some circumstances, but sonnet 3.5 was nearly instant.
Learning to Reason with LLMs
771–780 of 1001 posts
Re: Learning to Reason with LLMs
#772The "safety" example in the "chain-of-thought" widget/preview in the middle of the article is absolutely ridiculous. Take a step back and look at what OpenAI is saying here "an LLM giving detailed instructions on the synthesis of strychnine is unacceptable, here is what was previously generated vs our preferred, neutered content " What's this obsession with "safety" when it comes to LLMs? "This knowledge is perfectly…
tl;dr You can easily ask an LLM to return JSON results, and now working code, on your exact query and plug those to another system for automation. —- LLMs are usually accessible through easy-to-use API which can be used in an automated system without human in the loop. Larger scale and parallel actions with this method become far more plausible than traditional means. Text-to-action capabilities are powerful and gett…
Okay? And? What does that have to do with anything. I thought the number one rule of these things is to not trust their output?
If you are automatically formulating some chemical based on JSON results from ChatGPT and your building blows up… that is kind of on you.
Re: Learning to Reason with LLMs
#773Some practical notes from digging around in their documentation: In order to get access to this, you need to be on their tier 5 level, which requires $1,000 total paid and 30+ days since first successful payment. Pricing is $15.00 / 1M input tokens and $60.00 / 1M output tokens. Context window is 128k token, max output is 32,768 tokens. There is also a mini version with double the maximum output tokens (65,536 tokens…
I have access to this and there is no way I spend more than 50$ on OpenAI api. I have ChatGPT + since day q though (240$ probably in total)
You missed your raise key on "day q"
Re: Learning to Reason with LLMs
#774Can we please stop using the word “think” like o1 thinks before it answers. I doubt we man the same when someone says a human thinks vs o1 thinks. When I say I think “red” I am sure the word think means something completely different than when you say openai thinks red. I am not saying one is superior than the other but maybe as humans we can use a different set of terminology for the AI activities.
Re: Learning to Reason with LLMs
#775Student here. Can someone give me one reason why I should continue in software engineering that isn't denial and hopium?
What's the alternative? If AI is going to replace software engineers, there is no fundamental reason they couldn't replace almost all other knowledge workers as well. No matter the field, most of it is just office work managing, transforming and building new information, applying existing knowledge on new problems (that probably are not very unique in grand scheme of things). Except for medical doctors, nurses, and s…
Why can't medical doctors be automated?
Re: Learning to Reason with LLMs
#776Prompt: > Alice, who is an immortal robotic observer, orbits a black hole on board a spaceship. Bob exits the spaceship and falls into the black hole. Alice sees Bob on the edge of the event horizon, getting closer and closer to it, but from her frame of reference Bob will remain forever observable (in principle) outside the horizon. > > A trillion year has passed, and Alice observes that the black hole is now relati…
Out of curiosity, what answer to that would you find acceptable? I don't know relativity well enough to even speculate.
Re: Learning to Reason with LLMs
#777I’m not surprised there’s no comparison to GPT-4. Was 4o a rewrite on lower specced hardware and a more quantized model, where the goal was to reduce costs while trying to maintain functionality? Do we know if that is so? That’s my guess. If so is O1 an upgrade in reasoning complexity that also runs on cheaper hardware?
They call GPT4 a legacy model, maybe that's why they don't compare to it.
Re: Learning to Reason with LLMs
#778Earlier quoted context omitted.
Wow, that is impressive! How were you able to use o1-preview? I pay for ChatGPT, but on chatgpt.com in the model selector I only see 4o, 4o-mini, and 4. Is o1 in that list for you, or is it somewhere else?
Like others here, it was just available on the website and app when I checked. FWIW I still don’t have advanced voice mode.
I don’t have either the new model nor the advanced voice mode as a paying user.
Re: Learning to Reason with LLMs
#779Earlier quoted context omitted.
LLMs perform well on small tasks that are well defined. This definition matches almost every task that a student will work on in school leading to an overestimation of LLM capabiity. LLMs cannot decide what to work on, or manage large bodies of work/code easily. They do not understand the risk of making a change and deploying it to production, or play nicely in autonomous settings. There is going to be a massive amou…
Careers are 30 years long Can you confidently say that an LLM won’t be better than an average 22 year old coder within these 30 years?
> Can you confidently say that an LLM won’t be better than an average 22 year old coder within these 30 years?
No 22 years old coder is better than the open source library he's using taken straight from github, and yet he's the one who's getting paid for it.
People who claim IA will disrupt software development are just missing the big picture here: software jobs are already unrecognizable from what it was just 20 years ago. AI is just another tool, and as long as execs won't bother use the tool by themselves, then they'll pay developers to do it instead.
Over the past decades, writing code has become more and more efficient (better programming languages, better tooling, then enormous open source libraries) yet the number of developers kept increasing, it's Jevons paradox[1] in its purest form. So if past tells us anything, is that AI is going to create many new software developer jobs! (because the amount of people able to ship significant value to a customer is going to skyrocket, and customers' needs are a renewable resource).
Re: Learning to Reason with LLMs
#780I tried it with a cipher text that ChatGPT4o flailed with.
Recently I tried the same cipher with Claude Sonnet 3.5 and it solved it quickly and perfectly.
Just now tried with ChatGPT o1 preview and it totally failed. Based on just this one test, Claude is still way ahead.
ChatGPT also showed a comical (possibly just fake filler material) journey of things it supposedly tried including several rewordings of "rethinking my approach." It remarkably never showed that it was trying common word patterns (other than one and two letters) nor did it look for "the" and other "th" words nor did it ever say that it was trying to match letter patterns.
I told it upfront as a hint that the text was in English and was not a quote. The plaintext was one paragraph of layman-level material on a technical topic including a foreign name, text that has never appeared on the Internet or dark web. Pretty easy cipher with a lot of ways to get in, but nope, and super slow, where Claude was not only snappy but nailed it and explained itself.