Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

thebullshitmachines.com

Post reply on HN
Print view View original

341–350 of 652 posts

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#341

Post by bwfan123 »

maxdoop wrote: post #313
bwfan123 wrote: post #293

Earlier quoted context omitted.

the machine is fooling you with a mimicry of reasoning. and you are falling for it.

What is reasoning? What is understanding? Do humans do either? How do you know?

this is the question that the greeks wrestled with over 2000 years ago. at the time there were the sophists (modern llm equivalents) that could speak persuasively like a politician.

over time this question has been debated by philosophers, scientists, and anyone who wanted to have better cognition in general.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#342

Post by s2radhak »

Fascinating. The article repeatedly makes the claim that “LLMs work by predicting likely next words in a string of text”. Yet there’s the seemingly contradictory implication that we don’t know how LLMs work (ie we don’t know their secret sauce). How does one reconcile this? They’re either fancy autocompletes, or magic autocompletes (in which case the magic qualifier seems more important in understanding what they are than the autocomplete part).

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#343

Post by sabas123 »

myaccountonhn wrote: post #144
Johanx64 wrote: post #141

Somebody made a website to express their opinion - wherein their opinion can be surmised by reading the domain name. Text is scaled to 300% to indicate just how important and authoritative they think their opinion is. And it talks down to you in a "here comes the expert" style, with an atrocious aimed-at-preschoolers presentation. No thank you.

Two university professors in data science and computational biology are not just “somebody”.

Being a "University Professor" means jack shit unless precisely in their (sub)-field. The authors are experts in biology, and evolution of information representation/communication, and about misinformation.

I'll gladly defer to their expert opinion on those topics, but IMO to use such an authoritative voice when they are not experts in actual AI systems. Judging the massive progress in the field of AI, how can anyone even remotely state what these systems inherently are, when they are still so new and ever-evolving?

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#344

Post by llm_trw »

I thought I'd give this the benefit of the doubt.

It's trying its hardest to be put in the "this is bullshit" pile.

Five lines of content spread through five pages through the magic of parallax scrolling. Examples wandering around without going anywhere and confidently repeating talking points that were debunked 3 years ago.

Please release a textbook that can be read instead of whatever this is.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#345

Post by llm_trw »

pjs_ wrote: post #288

Not sure why everyone rates this. It’s full of very confidently made statements like “the AI has no ground truth” (obviously it does, it has ingested every paper ever), it “can’t reason logically” which seems like a stretch if you ever read the CoT of a frontier reasoning model and “can’t explain how they arrived at conclusions” where - I mean just try it yourself with o1, go as deep as you like asking how it arrived…

I've literally build a dynamic bench mark where I test reasoning models on their performance on deriving conclusions from assumptions through sequent calculus.

o3-mini high effort can derive chains that are 8 inference rules deep with >95% confidence I didn't have the money to test it further. This is better than the average professor in logic when given pen and paper.

It seems like a course critiquing 5 year old technology at this point.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#346

Post by lanstin »

nullc wrote: post #339
pjs_ wrote: post #288

Not sure why everyone rates this. It’s full of very confidently made statements like “the AI has no ground truth” (obviously it does, it has ingested every paper ever), it “can’t reason logically” which seems like a stretch if you ever read the CoT of a frontier reasoning model and “can’t explain how they arrived at conclusions” where - I mean just try it yourself with o1, go as deep as you like asking how it arrived…

> I mean just try it yourself with o1, go as deep as you like asking how it arrived at a conclusion I don't mean to disagree overall, but on this point the LLM can post-facto rationalize its output but it has no introspection and has absolutely no idea why it made a given bit of output (except in so far as it was a result of COT which it could reiterate to you). The set of weights being activated could be nearly disj…

I doubt people are very accurate at knowing why they made the choices they did. If you want them to recite a chain of reasoning they can but that is kind of far from most decision making most people do.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#347

Post by globalnode »

ctbergstrom wrote: post #307
grepLeigh wrote: post #303

Earlier quoted context omitted.

LLMs that use Chain of Thought sequences have been demonstrated to misrepresent their own reasoning [1]. The CoT sequence is another dimension for hallucination. So, I would say that an LLM capable of explaining its reasoning doesn't guarantee that the reasoning is grounded in logic or some absolute ground truth. I do think it's interesting that LLMs demonstrate the same fallibility of low quality human experts (i.e.…

Yes! First, thank you for the link about CoT misrepresentation. I've written a fair bit about this on Bluesky etc but I don't think much if any of that made it into the course yet. We should add this to lesson 6, "They're Not Doing That!" Your point about humanities courses is just right and encapsulates what we are trying to do. If someone takes the course and engages in the dialectical process and decides we are mu…

I'll give it a read. I must admit, the more I learn about the inner workings of LLM's the more I see them as simply the sum of their parts and nothing more. The rest is just anthropomorphism and marketing.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#348

Post by sabas123 »

neuronic wrote: post #48

> Moreover, a hallucination is a pathology. It's something that happens when systems are not working properly. > When an LLM fabricates a falsehood, that is not a malfunction at all. The machine is doing exactly what it has been designed to do: guess, and sound confident while doing it. > When LLMs get things wrong they aren't hallucinating. They are bullshitting. Very important distinction and again, shows the marke…

If you make an LLM which design goal is to state "I do not know" any answer that is not directly in its training set, then all of the above statements don't hold.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#349

Post by llm_trw »

nmca wrote: post #314
jdlshore wrote: post #300

Earlier quoted context omitted.

I read the whole course. Lesson 16, “The Next-Step Fallacy,” specifically addresses your argument here.

The discourse around synthetic data is like the discourse around trading strategies — almost anyone who really understands the current state of the art is massively incentivised not to explain it to you. This makes for piss-poor public epistemics.

I'm happy to explain my strategies about synthetic data - it's just that you'll need to hear about the onions I wore in my day: https://www.youtube.com/watch?v=yujF8AumiQo

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#350

Post by habinero »

nmca wrote: post #314
jdlshore wrote: post #300

Earlier quoted context omitted.

I read the whole course. Lesson 16, “The Next-Step Fallacy,” specifically addresses your argument here.

The discourse around synthetic data is like the discourse around trading strategies — almost anyone who really understands the current state of the art is massively incentivised not to explain it to you. This makes for piss-poor public epistemics.

Nah, you don't need to know the details to evaluate something. You need the output and the null hypothesis.

If a trading firm claims they have a wildly successful new strategy, for example, then first I want to see evidence they're not lying - they are actually making money when other people are not. Then I want to see evidence they're not frauds - it's easy to make money if you're insider trading. Then I want to see evidence that it's not just luck - can they repeat it on command? Then I might start believing they have something.

With LLMs, we have a bit of real technology, a lot of hype, a bunch of mediocre products, and people who insist if you just knew more of the secret details they can't explain, you'd see why it's about to be great.

Call it Habiñero's Razor, but for hype the most cynical explanation is most likely correct -- it's bullshit. If you get offended and DARVO when people call your product a "stochastic parrot", then I'm going to assume the description is accurate.

Post reply on HN