I think openai introduced the o1 model because reflection 70b inspired them. Getting them needed a new message to fill the gap for such a long time
Learning to Reason with LLMs
951–960 of 1001 posts
Re: Learning to Reason with LLMs
#952The "safety" example in the "chain-of-thought" widget/preview in the middle of the article is absolutely ridiculous. Take a step back and look at what OpenAI is saying here "an LLM giving detailed instructions on the synthesis of strychnine is unacceptable, here is what was previously generated vs our preferred, neutered content " What's this obsession with "safety" when it comes to LLMs? "This knowledge is perfectly…
I feel very alone in my view on caution and regulations here on HN. I am European and very happy we don't have the lax gun laws of the US. I also wished there had been more regulations on social media algorithms, as I feel that they have wreaked havoc on the society.
I guess it's just an ideological divide.
Re: Learning to Reason with LLMs
#953Reading through the Chain of Thought for the provided Cipher example (go to the example, click "Show Chain of Thought") is kind of crazy...it literally spells out every thinking step that someone would go through mentally in their head to figure out the cipher (even useless ones like "Hmm"!). It really seems like slowing down and writing down the logic it's using and reasoning over that makes it better at logic, simi…
Seriously. I actually feel as impressed by the chain of thought, as I was when ChatGPT first came out. This isn't "just" autocompletion anymore, this is actual step-by-step reasoning full of ideas and dead ends and refinement, just like humans do when solving problems. Even if it is still ultimately being powered by "autocompletion". But then it makes me wonder about human reasoning, and what if it's similar? Just fo…
An exception i came up with was from a documentary on einstein that described how he did his thought experiments. He would, of course, imagine novel scenarios in his head, which led him to the insights he could rephrase into language. I worry language models will still lack that capacity for insights driven by imagination.
Re: Learning to Reason with LLMs
#954Earlier quoted context omitted.
I suspect that actually reading the "chain of thought" would reveal obvious "logic" errors embarrassingly often.
It would still be auditable. In a few industries that is the only blocker for adoption--even if the outputs are incorrect.
I'll give an argument against this with the caveat it applies only if these are pure LLMs without heuristics or helper models (I do not believe that to be the case with o1).
The problem with auditing is not only are the outputs incorrect, but the "inputs" of the chained steps have no fundamental logical connection to the outputs. A statistical connection yes, but not a causal one.
For the trail to be auditable, processing would have to be taking place at the symbolic level of what the tokens represent in the steps. But this is not what happens. The transformer(s) (because these are now sampling multiple models) are finding the most likely set of tokens that reinforce a training objective which is a completed set of training chains. It is fundamentally operating below the symbolic or semantic level of the text.
This is why anthropomorphizing these is so dangerous. It isn't actually "explaining" its work. The CoT is essentially one large output, broken into parts. The RL training objective does two useful things: (1) break it down into much smaller parts, which drops the error significantly as that scales as an exponential of the token length, and (2) provides better coverage of training data for common subproblems. Both of those are valuable. Obviously, in many cases the reasons actually match the output. But hallucinations can happen anywhere throughout the chain, in ways which are basically undeterministic.
An intermediate step can provide a bad token and blithely ignore that to provide a correct answer. If you look at intermediate training of addition in pure LLMs, you'll get lots of results that look sort of like:
> "Add 123 + 456 and show your work"
> "First we add 6 + 3 in the single digits which is 9. Moving on we have 5 + 2 which is 8 in the tens place. And in the hundreds place, we have 5. This equals 579."
The above is very hand-wavy. I do not know if the actual prompts look like that. But there's an error in the intermediate step (5 + 2 = 8) that does not actually matter to the output. Lots of "emergent" properties of LLMs—arguably all of them—go away when partial credit is given for some of the tokens. And this scales predictably without a cliff [1]. This is also what you would expect if LLMs were "just" token predictors.
But if LLMs are really just token predictors, then we should not expect intermediate results to matter in a way in which they deterministically change the output. It isn't just that CoT can chaotically change future tokens, previous tokens can "hallucinate" in a valid output statement.
[1] Are Emergent Abilities of Large Language Models a Mirage?: https://arxiv.org/abs/2304.15004
Re: Learning to Reason with LLMs
#955This is great. I've been wondering how we will revert back to an agrarian society! You know, beating our swords into plowshares; more leisure time, visiting with good people, getting to know their thoughts hopes and dreams, playing music together, taking time contemplating the vastness and beauty of the universe. We're about to come full circle; back to Eden. It all makes sense now.
Is there a new drug we need to know about?
Re: Learning to Reason with LLMs
#956Why so much hate? They're doing their best. This is the state of progress in the field so far. The best minds are racing to innovate. The benchmarks are impressive nonetheless. Give them a break. At the end of the day, they built the chatbot who's saving your ass each day ever since.
Rarely using it at work, seems you are overestimating
Re: Learning to Reason with LLMs
#957The progress in AI is incredibly depressing, at this point I don't think there's much to look forward to in life. It's sad that due to unearned hubris and a complete lack of second-order thinking we are automating ourselves out of existence. EDIT: I understand you guys might not agree with my comments. But don't you thinking that flagging them is going a bit too far?
Eh this makes me very, very excited for the future. I want results, I don’t care if they come from humans or AI. That being said we might all be out of jobs soon…
Same thing hype bros told 2 years ago, won’t happen.
Re: Learning to Reason with LLMs
#958The model performance is driven by chain of thought, but they will not be providing chain of thought responses to the user for various reasons including competitive advantage. After the release of GPT4 it became very common to fine-tune non-OpenAI models on GPT4 output. I’d say OpenAI is rightly concerned that fine-tuning on chain of thought responses from this model would allow for quicker reproduction of their resu…
Am I right that this CoT is not actual reasoning in the same way that a human would reason, but rather just a series of queries to the model that still return results based on probabilities of tokens?
Re: Learning to Reason with LLMs
#959Great, yet another step towards the inevitable conclusion. Now I'm not just being asked to outsource my thinking to my computer, but instead to a black box operated by a for-profit company for the benefit of Microsoft. Not only will they not tell me the whole reasoning chain, they wont even tell me how they came up with it. Tell me, users of this tool. What's even are you? If you've outsourced your thinking to a corp…
> What's even are you? An accelerator of reaching the Singularity. This is something more.
You realize that you're not going inside the computer right? At best you're going to create a simulacrum of you. Something that looks, talks, and acts like you. It's never going to actually be you. You're going to be stuck out here with the rest of us, in whatever world we create in pursuit of the singularity suicide cult.
Re: Learning to Reason with LLMs
#960The model performance is driven by chain of thought, but they will not be providing chain of thought responses to the user for various reasons including competitive advantage. After the release of GPT4 it became very common to fine-tune non-OpenAI models on GPT4 output. I’d say OpenAI is rightly concerned that fine-tuning on chain of thought responses from this model would allow for quicker reproduction of their resu…
Am I right that this CoT is not actual reasoning in the same way that a human would reason, but rather just a series of queries to the model that still return results based on probabilities of tokens?
Tough question (for me). Assuming the model is producing its own queries, am I wrong to wonder how it's fundamentally different from human reasoning?