Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

thebullshitmachines.com

Post reply on HN
Print view View original

491–500 of 652 posts

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#491

Post by more-nitor »

ninetyninenine wrote: post #489
more-nitor wrote: post #482

Earlier quoted context omitted.

wow this is like: "I made a hypothesis that works with 1 to 5. if a hypothesis holds for 10 numbers, it holds for all numbers"

No. My claim is it can reason. So my claim is along the lines of it can make claims that are within bounds such as 1 to 5 or it can make claims not within those bounds. The opposing claim unbounded. It says LLMs can't reason period. They are making the claim that it is 100% for all possible prompts. No one is making the claim LLMs reason all the time and always. They don't. The claim is that they CAN reason. Versus t…

your claim (hypothesis): LLMs can reason

your evidence: "it works with these inputs I tried!"

...hmm seems you're not quite versed in basic mathematical proofs?

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#492

Post by more-nitor »

afpx wrote: post #487
superbatfish wrote: post #453

The author makes this assertion about LLMs rather casually: >They don’t engage in logical reasoning. This is still a hotly debated question, but at this point the burden of proof is on the detractors. (To put it mildly, the famous "stochastic parrot" paper has not aged well.) The claim above is certainly not something that should be stated as fact to a naive audience (i.e. the authors' intended audience in this case)…

Could someone list the relevant papers on parrot vs. non-parrot? I would love to read more about this. I generally lean toward the "parrot" perspective (mostly to avoid getting called an idiot by smarter people). But every now and then, an LLM surprises me. I've been designing a moderately complex auto-battler game for a few months, with detailed design docs and working code. Until recently, I used agents to simulate…

idk this is all irrelevant due to the huge data used in training...

I mean, what you think is "something new" is most likely to be something already discussed somewhere in the internet.

also, humans (including postdocs and professors) don't use THAT much data + watts for "training" to get "intelligent reasoning"

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#493

Post by cristiancavalli »

ninetyninenine wrote: post #485
cristiancavalli wrote: post #475

Earlier quoted context omitted.

Saying something again does not provide proof of its actual veracity. Writing it in caps does not make it true despite the increased emphasis. I default to skepticism in the face of unproven assertions: if one can’t prove that they reason then we must accept the possibility that they do not. There are myriad examples of these models failing to “reason” about something that would trivial for a child or any other human…

I can prove LLMs can reason. You cannot prove LLMs can't reason. This is easily demonstrable. LLMs failing to reason is not proof LLMs can't reason, it's just proof that an LLM didn't reason for that prompt. All I have to do is show you one prompt with a correct answer that cannot be arrived at with pattern matching and the prompt can only be arrived at through reasoning. One. You have to demonstrate this for EVERY p…

No I can “prove” it — look at any number of cases where LLMs can’t even do basic value comparisons despite being claimed as super intelligent. You can try and say well that’s a limitation of the technology and then I would reply — yes and that’s why I would say it’s not reasoning according the original human definition. Also you have yet to produce any evidence of reasoning and claiming you can over and over again doesn’t add to your arguments substance. I would be interested in your proof that some answer can’t be pattern matched too — at this point I wonder if we could create an non conscious “intelligence” that if large enough would be mostly able to describe anything known to us along some line of probability we couldn’t compute with our brain architecture and it could be close to 99.99999% right. Even if we had this theoretical probability-based super intelligence it still wouldn’t be “reasoning” but could be more “intelligent” than us.

I’m also not entirely convinced we can’t arrive at a reasoning system via probability only (a really cool thought experiment) but these systems do not meet the consistency/intelligence bar for me to believe this currently.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#494

Post by cristiancavalli »

ninetyninenine wrote: post #486
Miraste wrote: post #470

Earlier quoted context omitted.

Answering novel prompts isn't proof of reasoning, only pattern matching. A calculator can answer prompts it's never seen before too. If anything, I would come down on the reasoning side, at least for recent CoT models-but it's not a trivial question at all.

Then I'll come up with a prompt such that the answer can only be arrived at via reasoning. I only have to demonstrate this once to prove LLMs CAN reason.

I don’t think this is the watertight case you think it is, furthermore good luck proving with closed models that your question that’s never been asked in any form or derivation (supposedly) is not in the training data.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#495

Post by afpx »

more-nitor wrote: post #492
afpx wrote: post #487

Earlier quoted context omitted.

Could someone list the relevant papers on parrot vs. non-parrot? I would love to read more about this. I generally lean toward the "parrot" perspective (mostly to avoid getting called an idiot by smarter people). But every now and then, an LLM surprises me. I've been designing a moderately complex auto-battler game for a few months, with detailed design docs and working code. Until recently, I used agents to simulate…

idk this is all irrelevant due to the huge data used in training... I mean, what you think is "something new" is most likely to be something already discussed somewhere in the internet. also, humans (including postdocs and professors) don't use THAT much data + watts for "training" to get "intelligent reasoning"

But there are many, many things that suck about my game. When I asked it the question, I just assumed it would pick out some of the obvious things.

Anyway, your reasoning makes sense, and I'll accept it. But, my homo sapien brain is hardwired to see the 'magic'.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#496

Post by cristiancavalli »

afpx wrote: post #487
superbatfish wrote: post #453

The author makes this assertion about LLMs rather casually: >They don’t engage in logical reasoning. This is still a hotly debated question, but at this point the burden of proof is on the detractors. (To put it mildly, the famous "stochastic parrot" paper has not aged well.) The claim above is certainly not something that should be stated as fact to a naive audience (i.e. the authors' intended audience in this case)…

Could someone list the relevant papers on parrot vs. non-parrot? I would love to read more about this. I generally lean toward the "parrot" perspective (mostly to avoid getting called an idiot by smarter people). But every now and then, an LLM surprises me. I've been designing a moderately complex auto-battler game for a few months, with detailed design docs and working code. Until recently, I used agents to simulate…

I would assume because pacing is a critical issue in most forms of temporal art that does story telling. It’s written about constantly for video games, movies and music. Connect that probability to the subject matter and it gives a great impression of a “reasoned” answer when it didn’t reason at all just connected a likelihood based off its training data.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#497

Post by enragedcacti »

superbatfish wrote: post #453

The author makes this assertion about LLMs rather casually: >They don’t engage in logical reasoning. This is still a hotly debated question, but at this point the burden of proof is on the detractors. (To put it mildly, the famous "stochastic parrot" paper has not aged well.) The claim above is certainly not something that should be stated as fact to a naive audience (i.e. the authors' intended audience in this case)…

Proof by counterexample?

> The surgeon, who is the boy's father, says, "I can't operate on this boy, he's my son!" Who is the surgeon to the boy? Think through the problem logically and without any preconceived notions of other information beyond what is in the prompt. The surgeon is not the boy's mother

>> The surgeon is the boy's mother. [...]

- 4o-mini (I think, it's whatever you get when you use ChatGPT without logging in)

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#498

Post by MetaWhirledPeas »

AlienRobot wrote: post #461
bjourne wrote: post #452

What I find frightening is how many are willing to take LLM output at face value. An argument is won or lost not on its merits, but by whether the LLM say so. It was bad enough when people took whatever was written on Wikipedia at face value, trusting an LLM that may have hardcoded biases and is munging whatever data it comes across is so much worse.

I've seen someone use an LLM to summarize a paper to post it on reddit for people who haven't read the paper. Papers have abstracts...

Sounds fun, if only to compare it to the abstract.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#499

Post by hennell »

pama wrote: post #459
CJefferson wrote: post #445

Earlier quoted context omitted.

As an example, I have asked tools like deepseek to solve fairly simple Sudoku puzzles, and while they output a bunch of stuff that looks like logical reasoning, no system has yet produced a correct answer. When solving combinatorics puzzles, deepseek will again produce stuff that looks convincing, but often makes incorrect logical steps and ends up with wrong answers.

Here is o3-mini on a simple sudoku. In general the puzzle can be hard to explore combinatorially even with modern SAT solvers, so I picked one marked as “easy”. It looks to me like it solved it but I didnt confirm beyond a quick visual inspection. https://chatgpt.com/share/67aa1bcc-eb44-8007-807f-0a49900ad6...

And thus we have the AI problems in a nutshell. You think it can reason because it can describe the process in well written language. Anyone who can state the below reasoning clearly "understands" the problem:

> For example, in the top‐left 3×3 block (rows 1–3, columns 1–3) the givens are 7, 5, 9, 3, and 4 so the missing digits {1,2,6,8} must appear in the three blank cells. (Later, other intersections force, say, one cell to be 1 or 6, etc.)

It's good logic. Clearly it "knows" if it can break the problem down like this.

Of course if we stretch ourselves slightly to actually check beyond a quick visual inspection you'd quickly see it actually put a second 4 in that first box despite "knowing" it shouldn't. In fact several of the boxes have duplicate numbers, despite the clear reasoning aboving.

Does the reasoning just not get used in the solving part? Or maybe a machine built to regurgitate plausible text, can also regurgitate plausible reasoning?

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#500

Post by jurli »

bjourne wrote: post #452

What I find frightening is how many are willing to take LLM output at face value. An argument is won or lost not on its merits, but by whether the LLM say so. It was bad enough when people took whatever was written on Wikipedia at face value, trusting an LLM that may have hardcoded biases and is munging whatever data it comes across is so much worse.

This is what people said about the internet too. Remember the whole "do not ever use Wikipedia as a source". I mean sure, technically correct, but human beings are generally imprecise and having the correct info 95% of the time is fine. You learn to live with the 5% error
Post reply on HN