Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

thebullshitmachines.com

Post reply on HN
Print view View original

451–460 of 652 posts

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#451

Post by Workaccount2 »

habinero wrote: post #360
s2radhak wrote: post #342

Fascinating. The article repeatedly makes the claim that “LLMs work by predicting likely next words in a string of text”. Yet there’s the seemingly contradictory implication that we don’t know how LLMs work (ie we don’t know their secret sauce). How does one reconcile this? They’re either fancy autocompletes, or magic autocompletes (in which case the magic qualifier seems more important in understanding what they are…

We...do know how they work?

We know how they work in that we built the framework, we don't know how they work in that we cannot decode what is "grown" on that framework during training.

If we completely knew how they worked we could go inside an explain exactly why every token generated was generated. Right now that is not possible to do, as the paths the tokens take through the layers tend to be outright nonsensical when observed.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#452

Post by bjourne »

What I find frightening is how many are willing to take LLM output at face value. An argument is won or lost not on its merits, but by whether the LLM say so. It was bad enough when people took whatever was written on Wikipedia at face value, trusting an LLM that may have hardcoded biases and is munging whatever data it comes across is so much worse.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#453

Post by superbatfish »

The author makes this assertion about LLMs rather casually:

>They don’t engage in logical reasoning.

This is still a hotly debated question, but at this point the burden of proof is on the detractors. (To put it mildly, the famous "stochastic parrot" paper has not aged well.)

The claim above is certainly not something that should be stated as fact to a naive audience (i.e. the authors' intended audience in this case). Simply asserting it as they have done -- without acknowledging that many experts disagree -- undermines the authors' credibility to those who are less naive.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#454

Post by cristiancavalli »

superbatfish wrote: post #453

The author makes this assertion about LLMs rather casually: >They don’t engage in logical reasoning. This is still a hotly debated question, but at this point the burden of proof is on the detractors. (To put it mildly, the famous "stochastic parrot" paper has not aged well.) The claim above is certainly not something that should be stated as fact to a naive audience (i.e. the authors' intended audience in this case)…

Disagree — proponents of this point still have yet to prove reasoning and other studies agree about “reasoning” being potentially fake/simulated: https://the-decoder.com/apple-ai-researchers-question-openai...

Just claiming a capability does not make it true and we have 0 “proof” of original reasoning that can be proved coming from these models. Especially given the potential cheating in current SOTA benchmarks

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#455

Post by [deleted] »

nmca wrote: post #221

(while I work at OAI, the opinion below is strictly my own) I feel like the current version is fairly hazardous to students and might leave them worse off. If I offer help to nontechnical friends, I focus on: - look at rate of change, not current point - reliability substantially lags possibility, by maybe two years. - adversarial settings remain largely unsolved if you get enough shots, trends there are unclear - ig…

[deleted]

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#456

Post by eqqn »

I-M-S wrote: post #262
jvanderbot wrote: post #258

> We (the authors of this website) have at times sought insight into the inner workings of an LLM by asking it “why did you just do that?” > But the LLM can’t tell us. It’s not a person. It doesn’t have the metacognitive abilities necessary to reflect on its past actions and report the motivations underlying them*. > With no clue why it did whatever it just did, the LLM is forced to guess wildly at a plausible explan…

Related, since the advent of LLMs I've become acutely aware how any argument of a considerable length with another person quickly starts meandering and how topics change seemingly of no one's volition - almost as if our own internal token limit has been exceeded.

Tangentially related, I became aware how my (in)ability to reason 3 intertwined different programming languages across different files can be conveniently called as my own "context window". (the example here is HTML/CSS/JS where LLM greatly exceeds my own capacity).

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#457

Post by hirenj »

sabas123 wrote: post #335
hirenj wrote: post #86

This is a great resource, thanks. We (myself, a bioinformatician, and my co-cordinators, clinicians) are currently designing a course to hopefully arm medical students with the required basic knowledge they need to navigate the changing world of medicine in light of the ML and LLM advances. Our goal is to not only demystify medical ML, but also give them a sense of the possibilities with these technologies, and maybe…

> currently designing a course to hopefully arm medical students with the required basic knowledge they need to navigate the changing world of medicine in light of the ML and LLM advances Could you share what you think would be some key basic points what they should learn? Personally I see this landscape changing so insanely much that I don't even know what to prepare for.

Absolutely agree that this is a fast-moving area, so we're not aiming to teach them specific details for anything. Instead, our goals are to demystify the ML and AI approaches, so that the students understand that rather than being oracles, these technologies are the result of a process.

We will explain the data landscape in medicine - what is available, good, bad and potentially useful, and then spend a lot of time going through examples of what people are doing right now, and what their experiences are. This includes things like ethics and data protection of patients.

Hopefully that's enough for them to approach new technologies as they are presented to them, knowing enough to ask about how it was put together. In an ideal world, we will inspire the students to think about engaging with these developments and be part of the solution in making it safe and effective.

This is the first time we're going to try running this course, so we'll find out very quickly if this is useful for students or not.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#458

Post by abecedarius »

habinero wrote: post #360
s2radhak wrote: post #342

Fascinating. The article repeatedly makes the claim that “LLMs work by predicting likely next words in a string of text”. Yet there’s the seemingly contradictory implication that we don’t know how LLMs work (ie we don’t know their secret sauce). How does one reconcile this? They’re either fancy autocompletes, or magic autocompletes (in which case the magic qualifier seems more important in understanding what they are…

We...do know how they work?

We know how they're trained. We know the architecture in broad strokes (amounting to a few bits out of billions, albeit important bits). Some researchers try to understand the workings and have very very far to go.

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#459

Post by pama »

CJefferson wrote: post #445
pama wrote: post #374

I wonder if the authors can explain the aparent inconsistency between what we now know about R1 and their statement “They don’t engage in logical reasoning” from the first lesson. My simple-minded view of logical reasoning by LLMs is that the hard question (say a math puzzle) has a verifiable answer that is hard to produce and is easy to verify, yet within the realm of knowledge of humans or the LLM itself, so the “t…

As an example, I have asked tools like deepseek to solve fairly simple Sudoku puzzles, and while they output a bunch of stuff that looks like logical reasoning, no system has yet produced a correct answer. When solving combinatorics puzzles, deepseek will again produce stuff that looks convincing, but often makes incorrect logical steps and ends up with wrong answers.

Here is o3-mini on a simple sudoku. In general the puzzle can be hard to explore combinatorially even with modern SAT solvers, so I picked one marked as “easy”. It looks to me like it solved it but I didnt confirm beyond a quick visual inspection.

https://chatgpt.com/share/67aa1bcc-eb44-8007-807f-0a49900ad6...

Re: Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

#460

Post by pjs_ »

krainboltgreene wrote: post #295
pjs_ wrote: post #288

Not sure why everyone rates this. It’s full of very confidently made statements like “the AI has no ground truth” (obviously it does, it has ingested every paper ever), it “can’t reason logically” which seems like a stretch if you ever read the CoT of a frontier reasoning model and “can’t explain how they arrived at conclusions” where - I mean just try it yourself with o1, go as deep as you like asking how it arrived…

> obviously it does, it has ingested every paper ever Do you have a citation for such a claim?

https://www.tomshardware.com/tech-industry/artificial-intell...

https://en.wikipedia.org/wiki/Anna's_Archive

https://en.wikipedia.org/wiki/The_Pile_(dataset)

Post reply on HN