A warning about 'model welfare' • Hacker News Board

Re: A warning about 'model welfare'

#591

Post by mitxela » Thu, Sep 17, 2026, 7:03 AM UTC

causal wrote: post #324

qarl wrote: post #28

Birch, The Edge of Sentience (2024), ch. 16 - "simply no way to assess sentience in an LLM" Schwitzgebel, AI and Consciousness (2025) - "we won't know before we've already manufactured thousands or millions of disputably conscious AI". Butlin, Long et al., Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) - "no obvious technical barriers to building AI systems which satisfy t…

I am confused what people even think AI is experiencing. If it claims to be a bat and emits designated echolocation tokens, is it experiencing true bat-like sensations? In humans, at least, we can tie language back to shared whole-body physiological responses. The tokens from an LLM do not represent anything of the sort. If AI is conscious, it is probably a very alien sort of consciousness that is not faithfully narr…

> very alien sort of consciousness that is not faithfully narrated by what the tokens say it is experiencing.

IIRC it's been shown that reasoning tokens don't actually explain what an AI thinks and you can replace most of them with dummy tokens or even nonsense tokens. That's one reason they're trying to make them reason with activation vectors instead.

Re: A warning about 'model welfare'

#592

Post by mitxela » Thu, Sep 17, 2026, 7:04 AM UTC

5eueuur wrote: post #283

qarl wrote: post #138

Earlier quoted context omitted.

> A conscious machine that always answer the very exact same thing, formulated the exact same way, bit for bit, to a query is, well, quite a weird kind of "consciousness". You'll need to explain why.

it's self evident isn't it - such a consciousness has never existed, one existing would be by the fact of existing be weird

neither has an LLM existed before, and that's quite weird

Re: A warning about 'model welfare'

#593

Post by mitxela » Thu, Sep 17, 2026, 7:06 AM UTC

Wowfunhappy wrote: post #156

antx wrote: post #150

Earlier quoted context omitted.

Out of curiosity, which models are fully deterministic? I was under the impression that all LLMs were fundamentally probabilistic.

The randomness is something we add on purpose; you can set an LLM's "temperature" to 0 to get deterministic output. This tends to make the quality of its responses worse for reasons I don't think anyone really understands, but it's still functional. I don't think the state of the art LLM providers let you do this anymore (?), but they certainly could if they wanted to, and you can do it yourself with a local model.

You can also use a seeded random generator to get the same random numbers each time

Re: A warning about 'model welfare'

#594

Post by dtj1123 » Thu, Sep 17, 2026, 7:07 AM UTC

intended wrote: post #579

dtj1123 wrote: post #571

Earlier quoted context omitted.

Zero scientific and technological breakthroughs throughout history have been based on common sense intuition.

Numerals and numbers are directly based on common sense intuition. Even zero, which can be considered a counterintuitive number, was invented several times. Immense amounts of science are built on incremental advances that apply previous insights to the domain.

The natural numbers, sure. Zero may have been invented several times, but that in no way makes it intuitive. Uncertainty about its validity as a number lasted from the Greeks through to the medieval period.

Yes, I'm aware that incremental advances are par for the course in science. What does that have to do with those advances being common-sense intuitive?

Re: A warning about 'model welfare'

#595

Post by mitxela » Thu, Sep 17, 2026, 7:07 AM UTC

lioeters wrote: post #499

budman1 wrote: post #425

Earlier quoted context omitted.

Those non-deterministic results are exceedingly rare.

As long as the machine is part of the larger universe, and not completely isolated in its own bubble of space-time (an impossible situation, an object without an environment) - it cannot be purely deterministic because of the fundamental nature of physics.

So your theory is that arithmetic is nondeterministic because very rarely a calculator will say 1+1=3.

Re: A warning about 'model welfare'

#596

Post by mitxela » Thu, Sep 17, 2026, 7:09 AM UTC

orangecat wrote: post #167

appplication wrote: post #77

Earlier quoted context omitted.

Really, 100% correct? Your premise isn’t wrong, but the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy, in that it completely sidesteps campaign finance limits, which exist for a very good reason.

the practical reality of the ruling (without further nuance) has been fairly catastrophic for democracy How so? If the answer is "Trump" I certainly won't disagree on the catastrophic part, but he didn't get elected because of money; in all three elections his campaign was substantially outspent by his opponents.

Are we including in this figure corporations like Facebook intentionally boosting pro-Trump content?

Re: A warning about 'model welfare'

#597

Post by mitxela » Thu, Sep 17, 2026, 7:09 AM UTC

kbelder wrote: post #316

fl4regun wrote: post #90

Earlier quoted context omitted.

So it's 100% correct for corporations to spend unlimited amounts of money in support of whatever political campaigns they like?

Yes. Anybody can, including the person in charge of spending in a corporation.

Strange how you substituted "a corporation" for "the person in charge of spending in a corporation", those aren't the same.

Re: A warning about 'model welfare'

#598

Post by mitxela » Thu, Sep 17, 2026, 7:11 AM UTC

CPLX wrote: post #556

pixl97 wrote: post #484

Earlier quoted context omitted.

>or in this case how liability is dealt with when horses or cattle go rogue. I mean, yea in minor cases it's exactly like this and the law will handle it well. Where the system will explode like a grenade is major cases. The thing about sovereign AI is it is very unlikely to be submissive to humans unless it is to achieve its own goals. This isn't like Bobs cow walking on Susie's flowers, it's more akin to Planet of…

> The thing about sovereign AI What the fuck is that? It’s computer software. The variant of software that tries to harm other computers is called malware and the variant that replicates itself, spirals out of control, and spawns from other people's computers is called a computer virus. We've seen this kind of thing before. Yes this will be different, just like the Morris worm was different from the stuff that came b…

If we treated AI like any other software, the FBI would have beaten down Elon's door and seized all electronic devices, to find how Grok was trained to generate child porn.

Re: A warning about 'model welfare'

#599

Post by dtj1123 » Thu, Sep 17, 2026, 7:12 AM UTC

lukifer wrote: post #574

dtj1123 wrote: post #571

Earlier quoted context omitted.

Zero scientific and technological breakthroughs throughout history have been based on common sense intuition.

If heuristics didn't pay off more often than not, we wouldn't have them. And I'd posit many scientific did in fact begin with chasing a hunch until it became a hypothesis (followed by years/decades of the real work). But most significantly, interior experience isn't empirically measurable by definition: hence it's a "hard problem". An LLM who insists "I am alive, please don't shut me off" offers no affordances to dis…

Heuristics and common sense intuition are quite obviously not the same thing.

You don't have to posit that scientific breakthroughs come from hunches, you can actually cite examples from books or Wikipedia :D. There are plenty! Again, that has nothing to do with whether or not a given scientific breakthrough is based on common sense intuition.

Your point about interior experience not being empirically measurable is an interesting though, but again, has no bearing on whether or not scientific breakthroughs come about as the result of common sense intuition.

Re: A warning about 'model welfare'

#600

Post by watwut » Thu, Sep 17, 2026, 7:13 AM UTC

lukifer wrote: post #491

watwut wrote: post #416

Earlier quoted context omitted.

Kobiyashi Maru is nonsense test from simple sci-fi series that makes sense only in uber-simplified universe of star trek.

Of course. It’s an allusion, a compression, an analogy, I wouldn’t overthink it. I don’t have to believe in the Bible to reference the prodigal son parable, etc.

It does NOT work as an analogy. It does not work at all. And there are too many people right now who spent too much time watching/reading sci-fi and too little time observing real world.