Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

thebullshitmachines.com

Post reply on HN
Print view View original

471–480 of 652 posts

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#471

Post by wruza »

ninetyninenine wrote: post #468
cristiancavalli wrote: post #454

Earlier quoted context omitted.

Disagree — proponents of this point still have yet to prove reasoning and other studies agree about “reasoning” being potentially fake/simulated: https://the-decoder.com/apple-ai-researchers-question-openai... Just claiming a capability does not make it true and we have 0 “proof” of original reasoning that can be proved coming from these models. Especially given the potential cheating in current SOTA benchmarks

It’s stupid. You can prove that LLMs can reason by simply giving it a novel problem where no data exists and having it solve that problem. LLMs CAN reason. Whether it can’t reason is not provable. To prove that you have to give the LLM every possible prompt that it has no data for and effectively show it never reasons and gets it wrong all the time. Not only is the proof impossible but it’s already been falsified as…

You can prove that LLMs can reason by simply giving it a novel problem where no data exists and having it solve that problem

They scan a hyperdimensional problem space whose facetness and capacity a single human is unable to comprehend. But there potentially exist a slice that corresponds to a problem that is novel to a human. LLMs are completely alien to us both in capabilities and technicalities, so talking about whether they can reason makes as much sense as if you replaced “LLMs” with “rainforests” or “antarctica”.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#472

Post by cristiancavalli »

Miraste wrote: post #470
ninetyninenine wrote: post #468

Earlier quoted context omitted.

It’s stupid. You can prove that LLMs can reason by simply giving it a novel problem where no data exists and having it solve that problem. LLMs CAN reason. Whether it can’t reason is not provable. To prove that you have to give the LLM every possible prompt that it has no data for and effectively show it never reasons and gets it wrong all the time. Not only is the proof impossible but it’s already been falsified as…

Answering novel prompts isn't proof of reasoning, only pattern matching. A calculator can answer prompts it's never seen before too. If anything, I would come down on the reasoning side, at least for recent CoT models-but it's not a trivial question at all.

This is a fun thought experiment and made me reminisce on my Epistemology classes — something I think the current AI conversation would benefit greatly from. I’m super excited about what we’ve created here — less from the practical standpoint and more from a philosophical one where we get to interact with another form of distilled knowledge. It’s really too bad so much is breathless hype and grift because the philosophy student in me just wants to bask in thinking about this different form/medium/distillation of knowledge we now get to interact with. Comments like these help to reinvigorate that love though so thank you!

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#473

Post by Miraste »

CJefferson wrote: post #445
pama wrote: post #374

I wonder if the authors can explain the aparent inconsistency between what we now know about R1 and their statement “They don’t engage in logical reasoning” from the first lesson. My simple-minded view of logical reasoning by LLMs is that the hard question (say a math puzzle) has a verifiable answer that is hard to produce and is easy to verify, yet within the realm of knowledge of humans or the LLM itself, so the “t…

As an example, I have asked tools like deepseek to solve fairly simple Sudoku puzzles, and while they output a bunch of stuff that looks like logical reasoning, no system has yet produced a correct answer. When solving combinatorics puzzles, deepseek will again produce stuff that looks convincing, but often makes incorrect logical steps and ends up with wrong answers.

Then one has to ask: is it producing a facsimile of reasoning with no logic behind it, or is it just reasoning poorly?

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#474

Post by fritzo »

sgt101 wrote: post #66

I wish the title wasn't so aggressively anti-tech though. The problem is that I would like to push this course at work, but doing so would be suicidal in career terms because I would be seen as negative and disruptive. So the good message here is likely to miss the mark where it may be most needed.

What would be a better title? "Hallucinating" seems inaccurate. Maybe "Untrustworthy machines"? "Critical thinking"? "Street smarts for humans"? "Social studies including robots"?

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#475

Post by cristiancavalli »

ninetyninenine wrote: post #468
cristiancavalli wrote: post #454

Earlier quoted context omitted.

Disagree — proponents of this point still have yet to prove reasoning and other studies agree about “reasoning” being potentially fake/simulated: https://the-decoder.com/apple-ai-researchers-question-openai... Just claiming a capability does not make it true and we have 0 “proof” of original reasoning that can be proved coming from these models. Especially given the potential cheating in current SOTA benchmarks

It’s stupid. You can prove that LLMs can reason by simply giving it a novel problem where no data exists and having it solve that problem. LLMs CAN reason. Whether it can’t reason is not provable. To prove that you have to give the LLM every possible prompt that it has no data for and effectively show it never reasons and gets it wrong all the time. Not only is the proof impossible but it’s already been falsified as…

Saying something again does not provide proof of its actual veracity. Writing it in caps does not make it true despite the increased emphasis. I default to skepticism in the face of unproven assertions: if one can’t prove that they reason then we must accept the possibility that they do not. There are myriad examples of these models failing to “reason” about something that would trivial for a child or any other human (some are even given as examples in this posts other comments). Given this and the lack of concrete proof I currently tend to agree with the Apple researchers conclusion.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#476

Post by lsy »

pama wrote: post #374

I wonder if the authors can explain the aparent inconsistency between what we now know about R1 and their statement “They don’t engage in logical reasoning” from the first lesson. My simple-minded view of logical reasoning by LLMs is that the hard question (say a math puzzle) has a verifiable answer that is hard to produce and is easy to verify, yet within the realm of knowledge of humans or the LLM itself, so the “t…

I think part of this is that you can't trust the "thinking" output of the LLM to accurately convey what is going on internally to the LLM. The "thought" stream is just more statistically derived tokens based on the corpus. If you take the question "Is A a member of the set {A, B}?", the LLM doesn't internally develop a discrete representation of "A" as an object that belongs to a two-object set and then come to a distinct and absolute answer. The generated token "yes" is just the statistically most-likely next token that comes after those tokens in its corpus. And logical reasoning is definitionally not a process of "gaining confidence", which is all an LLM can really do so far.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#477

Post by radlad »

cristiancavalli wrote: post #472
Miraste wrote: post #470

Earlier quoted context omitted.

Answering novel prompts isn't proof of reasoning, only pattern matching. A calculator can answer prompts it's never seen before too. If anything, I would come down on the reasoning side, at least for recent CoT models-but it's not a trivial question at all.

This is a fun thought experiment and made me reminisce on my Epistemology classes — something I think the current AI conversation would benefit greatly from. I’m super excited about what we’ve created here — less from the practical standpoint and more from a philosophical one where we get to interact with another form of distilled knowledge. It’s really too bad so much is breathless hype and grift because the philoso…

Are there any good Epistemology resources online? Seems like we could all benefit from this these days.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#478

Post by Mistletoe »

bjourne wrote: post #452

What I find frightening is how many are willing to take LLM output at face value. An argument is won or lost not on its merits, but by whether the LLM say so. It was bad enough when people took whatever was written on Wikipedia at face value, trusting an LLM that may have hardcoded biases and is munging whatever data it comes across is so much worse.

I’d take the Wikipedia answer any day. Millions of eyes on each article vs. a black box with no eyes on the outputs.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#479

Post by radlad »

habinero wrote: post #338
layoric wrote: post #292

Earlier quoted context omitted.

How does this help the students with their use of these tools in the now, to not be left worse off? Most of the points you list seem like defending against criticism rather than helping address the harm.

Agree. It's also a virtue to point out the emperor has no clothes and the tailor peddling them is a bullshit artist. This is no different than the crypto people who insisted the blockchain would soon be revolutionary and used for everything, when in reality the only real use case for a blockchain is cryptocoins, and the only real use case for cryptocoins is crime. The only really good use case for LLMs is spam, becau…

> The only really good use case for LLMs is spam, because it's the only use case for generating a lot of human-like speech without meaning.

As someone who's been writing code for nearly 20 years now, and who spent a few weeks rewriting a Flutter app in Jetpack Compose with some help from Claude (https://play.google.com/store/apps/details?id=me.johnmaguire...), I have to say I don't agree with this at all.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#480

Post by hnthrow90348765 »

cristiancavalli wrote: post #454
superbatfish wrote: post #453

The author makes this assertion about LLMs rather casually: >They don’t engage in logical reasoning. This is still a hotly debated question, but at this point the burden of proof is on the detractors. (To put it mildly, the famous "stochastic parrot" paper has not aged well.) The claim above is certainly not something that should be stated as fact to a naive audience (i.e. the authors' intended audience in this case)…

Disagree — proponents of this point still have yet to prove reasoning and other studies agree about “reasoning” being potentially fake/simulated: https://the-decoder.com/apple-ai-researchers-question-openai... Just claiming a capability does not make it true and we have 0 “proof” of original reasoning that can be proved coming from these models. Especially given the potential cheating in current SOTA benchmarks

>Disagree — proponents of this point still have yet to prove reasoning and other studies agree about “reasoning” being potentially fake/simulated: https://the-decoder.com/apple-ai-researchers-question-openai...

???

https://the-decoder.com/language-models-use-a-probabilistic-...

Post reply on HN