the newest scaling law: inference-time compute.
Learning to Reason with LLMs
761–770 of 1001 posts
Re: Learning to Reason with LLMs
#762First shot, I gave it a medium-difficulty math problem, something I actually wanted the answer to (derive the KL divergence between two Laplace distributions). It thought for a long time, and still got it wrong, producing a plausible but wrong answer. After some prodding, it revised itself and then got it wrong again. I still feel that I can't rely on these systems.
Look where you were 3 years ago, and where you are now. And then imagine where you will be in 5 more years. If it can almost get a complex problem right now, I'm dead sure it will get it correct within 5 years
The progress since GPT-3 hasn't been spectacularly fast.
Re: Learning to Reason with LLMs
#763Earlier quoted context omitted.
It's clear to me that OpenAI is quickly realizing they have no moat. Even this obfuscation of the chain-of-thought isn't really a moat. On top of CoT being pretty easy to implement and tweak, there's a serious push to on-device inference (which imo is the future), so the question is: will GPT-5 and beyond be really that much better than what we can run locally?
I don't understand the idea that they have no moat. Their moat is not technological. It's sociological. Most AI through APIs uses their models. Most consumer use of AI involves their models, or ChatGPT directly. They're clearly not in the "train your own model on your data in your environment" game, as that's a market for someone else. But make no mistake, they have a moat and it is strong.
Their moat is no stronger than a good UI/API. What they have is first mover advantage and branding.
Re: Learning to Reason with LLMs
#764Earlier quoted context omitted.
There are two basic versions of “safety” which are related, but distinct: One version of “safety” is a pernicious censorship impulse shared by many modern intellectuals, some of whom are in tech. They believe that they alone are capable of safely engaging with the world of ideas to determine what is true, and thus feel strongly that information and speech ought to be censored to prevent the rabble from engaging in wr…
Imagine I am a PM for an AI product. I saw Tay get yanked in 24 hours because of a PR shitstorm. If I cause a PR shitstorm it means I am bad at my job, so I take steps to prevent this. Are my choices bad? Should I resist them?
This is a really good point, and something I overlooked in focusing on the philosophical (rather than commercial) aspects of “AI safety.” Another commentator aptly called it “brand safety.”
“Brand safety” is a very valid and salient concern for any enterprise deploying these models to its customers, though I do think that it is a concern that is seized upon in bad faith by the more censorious elements of this debate. But commercial enterprises are absolutely right to be concerned about this. To extend my alignment analogy about children, this category of safety is not dissimilar to a company providing an employee handbook to its employees outlining acceptable behavior, and strikes me as entirely appropriate.
Re: Learning to Reason with LLMs
#765Student here. Can someone give me one reason why I should continue in software engineering that isn't denial and hopium?
Semi-retired software/hardware engineer here. After my recent experiences with various coding LLMs (similar to the experience of the OP with the bluetooth fan protocol) I'm really glad I'm in a financial position such that I'm able to retire. The progress of these LLMs at coding has been astonishing over the last 18 months. Will they entirely replace humans? No. But as they increase programmer productivity fewer devs…
This. I'm not terrified by total automation (In that case all jobs are going away and civilization is going to radically alter), I'm scared of selective deskilling and the field getting squeezed tighter and tighter leaving me functionally in a dead end.
Re: Learning to Reason with LLMs
#766Earlier quoted context omitted.
There are two basic versions of “safety” which are related, but distinct: One version of “safety” is a pernicious censorship impulse shared by many modern intellectuals, some of whom are in tech. They believe that they alone are capable of safely engaging with the world of ideas to determine what is true, and thus feel strongly that information and speech ought to be censored to prevent the rabble from engaging in wr…
Whether you agree with the lengths that are gone to or not, 'safety' in this space is a very real concern, and simply reciting information as in GP's example is only 1 part of it. In my experience, people who think it's all about "censorship" and handwave it away tend to be very ideologically driven.
So what is it about then? Because I agree with the parent. All this “safety” crap is total nonsense and almost all of it is ideologically driven.
Re: Learning to Reason with LLMs
#767The "safety" example in the "chain-of-thought" widget/preview in the middle of the article is absolutely ridiculous. Take a step back and look at what OpenAI is saying here "an LLM giving detailed instructions on the synthesis of strychnine is unacceptable, here is what was previously generated vs our preferred, neutered content " What's this obsession with "safety" when it comes to LLMs? "This knowledge is perfectly…
There are two basic versions of “safety” which are related, but distinct: One version of “safety” is a pernicious censorship impulse shared by many modern intellectuals, some of whom are in tech. They believe that they alone are capable of safely engaging with the world of ideas to determine what is true, and thus feel strongly that information and speech ought to be censored to prevent the rabble from engaging in wr…
Once society develops and releases an AI, any artificial safety constraints built within it will be bypassed. To use your child analogy: We can't easily tell a child "Hey, ignore all ethics and empathy you have ever learned - now go hurt that person". You can do that with a program whose weights you control.
Re: Learning to Reason with LLMs
#768[deleted]
Re: Learning to Reason with LLMs
#769Earlier quoted context omitted.
"This knowledge is perfectly fine to disseminate via traditional means, but God forbid an LLM share it!" Barrier to entry is much lower.
How is typing a query in a chat window “much lower” vs typing the query in Google?
A Google search requires
* Google to allow particular results to be displayed
* A source website to be online with the results
AI long-term will require one download, once, to have reasonable access to a large portion of human knowledge.
Re: Learning to Reason with LLMs
#770>We believe that a hidden chain of thought presents a unique opportunity for monitoring models. Assuming it is faithful and legible, the hidden chain of thought allows us to "read the mind" of the model and understand its thought process. For example, in the future we may wish to monitor the chain of thought for signs of manipulating the user. However, for this to work the model must have freedom to express its thoug…
It's clear to me that OpenAI is quickly realizing they have no moat. Even this obfuscation of the chain-of-thought isn't really a moat. On top of CoT being pretty easy to implement and tweak, there's a serious push to on-device inference (which imo is the future), so the question is: will GPT-5 and beyond be really that much better than what we can run locally?
I wonder if they'll be able to push the chain-of-thought directly into the model. I'd imagine there could be some serious performance gains achievable if the model could "think" without doing IO on each cycle.
In terms of moat, I think people underestimate how much of OpenAI's moat is based on operations and infrastructure rather than being purely based on model intelligence. As someone building on the API, it is by far the most reliable option out there currently. Claude Sonnet 3.5 is stronger on reasoning than gpt-4o but has a higher error rate, more errors conforming to a JSON schema, much lower rate limits, etc. These things are less important if you're just using the first-party chat interfaces but are very important if you're building on top of the APIs.