Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ML promises to be profoundly weird

aphyr.com

Post reply on HN
Print view View original

361–370 of 641 posts

Re: ML promises to be profoundly weird

#361

Post by steveklabnik »

sillysaurusx wrote: post #355
GeoAtreides wrote: post #352

Earlier quoted context omitted.

he doesn't have solid points, he conflates fair use with free use (?), ignores thousands of years of attribution history, and equates normal human to human learning with corporate LLMs training on original content (without consent). Great presentation, like you said, to cover the logical defects.

Fair use of training data hasn’t yet been settled in court. People here are treating it like it has been. But no amount of wishful thinking or moral arguments will change a verdict saying it’s fine for training data to be used as it has been. Until that question is settled, it’s disingenuous to dismiss his points out of hand as conflating fair use or ignoring consent.

Even beyond that, the initial legal opinion we do have did in fact point to training being fair use: https://www.reuters.com/legal/litigation/anthropic-wins-key-...

However, I don't feel comfortable suggesting that this is settled just yet, one district judge's opinion does not mean that other future cases may disagree, or we may at some point get explicit legislation one way or the other.

Re: ML promises to be profoundly weird

#362

Post by simianwords »

camgunz wrote: post #358
simianwords wrote: post #340

Earlier quoted context omitted.

Specifically in the case where it can use tools - no it doesn't hallucinate. Which is why you are struggling to find counterexamples.

> Specifically in the case where it can use tools - no it doesn't hallucinate. OpenAI's own system card says it does. Hallucination rates in GPT-5 with browsing enabled: - 0.7% in LongFact-Concepts - 0.8% in LongFact-Objects - 1.0% in FActScore > Which is why you are struggling to find counterexamples. Hey look, over 500 counterexamples: [1]. GPT-5.4's hallucination rate on AA-Omniscience is 89% [0], which is atrocio…

You had to go all the way and find it in the benchmark results that specifically stress test this.

You could not come up with a single one yourself. And you also linked an example where it was not allowed to use tools when I specifically said that it should be able to use tools. I'm not sure why are you present this as though it is a big gotcha.

I think my main point pretty much stands.

Re: ML promises to be profoundly weird

#363

Post by GeoAtreides »

sillysaurusx wrote: post #355
GeoAtreides wrote: post #352

Earlier quoted context omitted.

he doesn't have solid points, he conflates fair use with free use (?), ignores thousands of years of attribution history, and equates normal human to human learning with corporate LLMs training on original content (without consent). Great presentation, like you said, to cover the logical defects.

Fair use of training data hasn’t yet been settled in court. People here are treating it like it has been. But no amount of wishful thinking or moral arguments will change a verdict saying it’s fine for training data to be used as it has been. Until that question is settled, it’s disingenuous to dismiss his points out of hand as conflating fair use or ignoring consent.

I was just enumerating some of the issues with the '''solid''' points OP made. Actually addressing them would take too long and be exercise in futility, here, in HN, in april 2026. Why would I put in the effort, for my comment to be flagged and sent to the void? or worse, persisted for ever and used for training without my consent?

And yes, you are right, the legal and moral question of fair use in training data hasn't been settled yet; we agree here.

Re: ML promises to be profoundly weird

#364

Post by simianwords »

simoncion wrote: post #356
simianwords wrote: post #325

Earlier quoted context omitted.

Haha ok. So still no example? The GPT shared link shows a "thought for" which indicates using the latest thinking model. You may try that. What you can do is this: submit a prompt that clearly makes GPT hallucinate. You may secretly use a worse model. You may use a system prompt that deliberately gives wrong answers. But I'm going to assume you won't go that far. We can leave it to the public to decide whether this i…

You weren't paying much attention to the "Consider:" part of my previous comment. You don't believe that a well-paid, very careful, high-integrity member of the computer safety community has -on multiple occasions- encountered actual, sustained bullshiting from the latest-available for-pay version of ChatGPT. You don't accept either this fellow's reports or my informed assessment of his computing situation as truthfu…

> On top of that, your goalpost-shifting and general demeanor throughout this conversation simply don't give me the impression that you've much integrity. I'm not spending the equivalent of ten-to-twenty six-packs to reproduce aphyr's work and -given the evidence I have before me- have you reject that, as well.

Lol what goal post did I move? I said text only and you rejected it. You can present the example here and let the public judge it - even if my integrity is compromised. I'm allowing you to do it.

> 200 USD is a lot of money to throw away to "win" an Internet argument with a stranger who refuses to accept evidence presented by someone known to be careful, scrupulous, and honest.

200 what? I'm using the $20 one. This is getting ridiculous!

You can't present a _single_ counter example!

Re: ML promises to be profoundly weird

#365

Post by gritspants »

satvikpendem wrote: post #350
gritspants wrote: post #336

Earlier quoted context omitted.

I don't think you understand. Frankly, AI is a failure if all it does is replace coders. AI needs (given its current investment levels) to conquer all forms of knowledge work. This is an example of tech/industry needing to impose itself on society, rather than society needing it.

That's how human progress works. No one can want or need it because they cannot conceptualize wanting it until someone shows that it is possible. Now, many of those wants become needs.

We can absolutely conceptualize what we want or need. I was born in 1980 in NYC. When I was a boy my father took me to a tech conference where they had a demo of ordering TV shows on demand. It was a miracle, to my young mind. Was this what I needed?

Growing up I had a friend group of misfit boys, who discovered h4ck1ng and phr34king. But we also discovered slackware Linux on 3.5" floppies. We also had to discover ASM and compiling the linux kernel in order to do anything with it. Boys with machines. That wasn't what I needed either.

Later on we did have great things with tech. Google made the world searchable in ways Altavista didn't. I remember strapping the original iPod on my arm to go for runs outside. I didn't even need a car for a while investors subsidized my Uber rides to and from the office.

Now, it seems the US is balanced on a precipice. The economy seems to have an incredible amount of money desperate to grow, but to what purpose. In my lifetime, and in my parents, and their parents before them, when the dollar becomes restless the flag goes forth. The dollar follows the flag.

And here we are at war.

Re: ML promises to be profoundly weird

#366

Post by kgeist »

_dwt wrote: post #58

I have a question for all the "humans make those mistakes too" people in this thread, and elsewhere: have you ever read, or at least skimmed a summary of, "The Origin of Consciousness in the Breakdown of the Bicameral Mind"? Did you say "yeah, that sounds right"? Do you feel that your consciousness is primarily a linguistic phenomenon? I am not trying to be snarky; I used to think that intelligence was intrinsically…

>and while I agree humans can make similar mistakes/confabulations, I overwhelmingly feel that there is no "there" there.

What really opened my eyes a couple weeks ago (anyone can try this): I asked Sonnet to write an inference engine for Qwen3, from scratch, without any dependencies, in pure C. I gave it GGUF specs for parsing (to quickly load existing models) and Qwen3's architecture description. The idea was to see the minimal implementation without all the framework fluff, or abstractions. Sonnet was able to one-shot it and it worked.

And you know what, Qwen3's entire forward pass is just 50 lines of very simple code (mostly vector-matrix multiplications).

The forward pass is only part of the story; you just get a list of token probabilities from the model, that is all. After the pass, you need to choose the sampling strategy: how to choose the next token from the list. And this is where you can easily make the whole model much dumber, more creative, more robotic, make it collapse entirely by just choosing different decoding strategies. So a large part of a model's perceived performance/feel is not even in the neurons, but in some hardcoded manually-written function.

Then I also performed "surgery" on this model by removing/corrupting layers and seeing what happens. If you do this excercise, you can see that it's not intelligence. It's just a text transformation algorithm. Something like "semantic template matcher". It generates output by finding, matching and combining several prelearned semantic templates. A slight perturbation in one neuron can break the "finding part" and it collapases entirely: it can't find the correct template to match and the whole illusion of intelligence breaks. Its corrupted output is what you expect from corrupting a pure text manipulation algorithm, not a truly intelligent system.

Re: ML promises to be profoundly weird

#367

Post by GeoAtreides »

steveklabnik wrote: post #359
GeoAtreides wrote: post #352

Earlier quoted context omitted.

he doesn't have solid points, he conflates fair use with free use (?), ignores thousands of years of attribution history, and equates normal human to human learning with corporate LLMs training on original content (without consent). Great presentation, like you said, to cover the logical defects.

I did say "free use" instead of "fair use," yeah. That's my mistake, thank you for the correction. If I could edit my original comment, I would, mea culpa. Typos happen.

I see. I must congratulate you on your rhetorical prowess, it's nice seeing a professional at work.

Re: ML promises to be profoundly weird

#368

Post by camgunz »

simianwords wrote: post #362
camgunz wrote: post #358

Earlier quoted context omitted.

> Specifically in the case where it can use tools - no it doesn't hallucinate. OpenAI's own system card says it does. Hallucination rates in GPT-5 with browsing enabled: - 0.7% in LongFact-Concepts - 0.8% in LongFact-Objects - 1.0% in FActScore > Which is why you are struggling to find counterexamples. Hey look, over 500 counterexamples: [1]. GPT-5.4's hallucination rate on AA-Omniscience is 89% [0], which is atrocio…

You had to go all the way and find it in the benchmark results that specifically stress test this. You could not come up with a single one yourself. And you also linked an example where it was not allowed to use tools when I specifically said that it should be able to use tools. I'm not sure why are you present this as though it is a big gotcha. I think my main point pretty much stands.

I found over 500 examples that fit your criteria. Embarrassing you were arguing in bad faith this whole time.

Re: ML promises to be profoundly weird

#369

Post by simianwords »

camgunz wrote: post #368
simianwords wrote: post #362

Earlier quoted context omitted.

You had to go all the way and find it in the benchmark results that specifically stress test this. You could not come up with a single one yourself. And you also linked an example where it was not allowed to use tools when I specifically said that it should be able to use tools. I'm not sure why are you present this as though it is a big gotcha. I think my main point pretty much stands.

I found over 500 examples that fit your criteria. Embarrassing you were arguing in bad faith this whole time.

They all use the tool search, no? Please correct me if I'm wrong.

My criteria was using ChatGPT which explicitly allows it.

https://arxiv.org/html/2511.13029v1 if you don't believe me.

BTW this was your original point

>Anyway, it's trivial to get pretty much any model to make things up. Don't we all know this? That's why I was surprised by your position; if we know anything about these things it's that they make things up.

And look at how much effort you have had to do

1. use the wrong model for the horns example

2. the game one also didn't work

3. now you are searching for examples in literal benchmarks and you are still not able to find any

How is this trivial in any interpretation of the word?

I think it would be perfectly reasonable to agree that it is not at all trivial to find counter examples for my challenge.

Re: ML promises to be profoundly weird

#370

Post by Retric »

munificent wrote: post #150

There is a whole giant essay I probably need to write at some point, but I can't help but see parallels between today and the Industrial Revolution. Prior to the industrial revolution, the natural world was nearly infinitely abundant. We simply weren't efficient enough to fully exploit it. That meant that it was fine for things like property and the commons to be poorly defined. If all of us can go hunting in the woo…

> Prior to the industrial revolution, the natural world was nearly infinitely abundant. We simply weren't efficient enough to fully exploit it.

This is just wildly incorrect. People started running out of trees during the early Iron Age. Woodlands have been a managed and often over exploited resource for a long time. Active agriculture vs passive woodlands vs animal grazing has been in constant tension for thousands of years across most of the globe.

Post reply on HN