Where the goblins came from • Hacker News Board

Re: Where the goblins came from

#661

Post by slopinthebag » Fri, May 01, 2026, 12:14 AM UTC

jeremyjh wrote: post #659

slopinthebag wrote: post #658

Earlier quoted context omitted.

It's not, and unfortunately you cannot just dismiss perhaps the greatest refutation of functionalism as being incoherent, you have to actually address the argument. Take a person (Fred) with no experience of knowledge of chess. They don't know how the game works, how the pieces move, or any of the rules. They memorise an algorithm, say how Leela does its search and evaluation, and they can then look at a position on…

There is no algorithm that can be memorized. Leela's understanding is in the weights, not neural net algorithms. I'm familiar with the Chinese room argument and I've never accepted it because what it describes isn't real. It imagines some algorithm for which there is no evidence. Show me this process running and then ask me if it understands Chinese. To me this is as philosophically dubious as the notion of p-zombies…

You can actually do calculations of LLMs and models like Leela on paper or in your head if you had enough time (and patience)! It's basically just a whole lotta matrix multiplication. It's a thought experiment and its validity does not rest on the ability for someone to actually do these calculations in a suitable timespan. The specifics of the algorithm have no relevance.

If you did see the process running, when asked would you say it understands Chinese?

Re: Where the goblins came from

#662

Post by jeremyjh » Fri, May 01, 2026, 12:25 AM UTC

slopinthebag wrote: post #661

jeremyjh wrote: post #659

Earlier quoted context omitted.

There is no algorithm that can be memorized. Leela's understanding is in the weights, not neural net algorithms. I'm familiar with the Chinese room argument and I've never accepted it because what it describes isn't real. It imagines some algorithm for which there is no evidence. Show me this process running and then ask me if it understands Chinese. To me this is as philosophically dubious as the notion of p-zombies…

You can actually do calculations of LLMs and models like Leela on paper or in your head if you had enough time (and patience)! It's basically just a whole lotta matrix multiplication. It's a thought experiment and its validity does not rest on the ability for someone to actually do these calculations in a suitable timespan. The specifics of the algorithm have no relevance. If you did see the process running, when ask…

You can do a thought experiment about an invisible pink dragon, that doesn't mean I have to take a position on it. "Suppose" is doing all the lifting. My position is that experiment can't happen as described.

There is no algorithm for manipulating abstract symbols in a manner that "speaks chinese" without "understanding" it. The experiment bakes in the conclusion from the beginning.

Re: Where the goblins came from

#663

Post by fud101 » Fri, May 01, 2026, 12:42 AM UTC

libraryofbabel wrote: post #571

therobots927 wrote: post #560

Earlier quoted context omitted.

I wouldn’t use the phrase “emergent behavior” when talking about a model trained on a larger dataset. The model is designed to learn statistical patterns from that data - of course giving it more data allows it to learn higher level patterns of language and apparent “reasoning ability”. I don’t think there’s anything mysterious going on. That’s why I said we understand how LLMs work. We may not know exactly how they’…

So, to reiterate my example: you'd have been fine with people claiming in 2019 that we would eventually scale LLMs to the capabilities of Opus 4.7 + Claude Code? Because I would have said then that was a fantasy, because "LLMs are just statistical pattern matchers." But I was wrong and I changed my opinion. (Or do you not think the current SoTA LLMs are impressive? If so I can't help you and this discussion won't go…

Will the RLVR mechanism be improved upon or is it in some sense optimal?

Re: Where the goblins came from

#664

Post by fud101 » Fri, May 01, 2026, 12:43 AM UTC

theowaway wrote: post #402

keybored wrote: post #368

Earlier quoted context omitted.

But here’s the realization I had. And it’s a serious thing. At first I was both saying that this intelligence was the most awesome thing put on the table since sliced bread and stoking fear about it being potentially malicious. Quite straightforwardly because both hype and fear was good for my LLM stocks. But then something completely unexpected happened. It asked me on a date. This made no sense. I had configured th…

I think you need to go outside and touch some grass

He's clearly mimicking one of the clankers.

Re: Where the goblins came from

#665

Post by airstrike » Fri, May 01, 2026, 12:47 AM UTC

nearbuy wrote: post #622

airstrike wrote: post #486

Earlier quoted context omitted.

That all of their model outputs should be influenced by whatever personality prompt voodoo the wise artisan at OpenAI decided to stuff it with during RL should give everyone pause. That Nerdy personality prompt made me gag. As a card-carrying Nerd, I feel offended

Just to clarify, it's not the prompt voodoo that caused the affinity for goblins. It's the reward. They rewarded it for mentioning goblins when set to Nerdy, and it's still the same model as the other personalities, so the effects can carry over.

Makes sense, but I don't know why they'd let said prompt voodoo touch RL. I'm OK with prompting to get the model to, I don't know, write better Rust or build Excel spreadsheets. I am less OK with making it "quirky" or having some "personality" in a way that becomes ingrained in the model for everyone else

TL;DR the cringe nerdy shit should be (optionally) switched on at inference, not as part of RL

Re: Where the goblins came from

#666

Post by slopinthebag » Fri, May 01, 2026, 1:03 AM UTC

jeremyjh wrote: post #662

slopinthebag wrote: post #661

Earlier quoted context omitted.

You can actually do calculations of LLMs and models like Leela on paper or in your head if you had enough time (and patience)! It's basically just a whole lotta matrix multiplication. It's a thought experiment and its validity does not rest on the ability for someone to actually do these calculations in a suitable timespan. The specifics of the algorithm have no relevance. If you did see the process running, when ask…

You can do a thought experiment about an invisible pink dragon, that doesn't mean I have to take a position on it. "Suppose" is doing all the lifting. My position is that experiment can't happen as described. There is no algorithm for manipulating abstract symbols in a manner that "speaks chinese" without "understanding" it. The experiment bakes in the conclusion from the beginning.

> My position is that experiment can't happen as described.

Say you are the room and are passed symbols on paper, like the suits of playing cards. You use a book (lookup table) to transform series of symbols into a new symbol, and pass it out of the room to the observer.

You get passed ♠ + ♣ and you return ♢. Do you have an understanding of the underlying concept? If so, reply and tell me what it is! But if you don't know what the underlying concept is, how could you argue that the person in the room does?

Re: Where the goblins came from

#667

Post by jibal » Fri, May 01, 2026, 1:29 AM UTC

keybored wrote: post #565

jibal wrote: post #388

Earlier quoted context omitted.

https://finmasters.com/ad-hoc-fallacy/ > Ad hoc fallacy is a fallacious rhetorical strategy in which a person presents a new explanation – that is unjustified or simply unreasonable – of why their original belief or hypothesis is correct after evidence that contradicts the previous explanation has emerged. https://cerebralfaith.net/logical-fallacy-series-part-13-ad-... > An argument is ad hoc if its only given in an…

Thanks. I’m by default disposition suspicious of fallacies that are not logical fallacies. And I’m not convinced that this is a solid fallacy. > > Ad hoc fallacy is a fallacious rhetorical strategy in which a person presents a new explanation – that is unjustified or simply unreasonable – of why their original belief or hypothesis is correct after evidence that contradicts the previous explanation has emerged. That s…

I only gave the definition from cerebralfaith ... I didn't read their example, which I agree is bogus. My mistake for including that reference without reading the rest.

> I’m by default disposition suspicious of fallacies that are not logical fallacies.

You mean formal fallacies. Informal fallacies like ad hoc are still logical fallacies.

> divine that the argumenter is intending to be dishonest

The intent is obvious when someone keeps inventing some new argument when their previous one is shown to be erroneous--they are attached to the conclusion, not guided by truthseeking. But divining intent isn't a necessity ... the process is not logically valid.

Re: Where the goblins came from

#668

Post by chongli » Fri, May 01, 2026, 2:15 AM UTC

jychang wrote: post #61

chongli wrote: post #42

Earlier quoted context omitted.

We’re really not that vulnerable to such things as a species, because we as individuals all have our own minds and our own sets of biases that cancel out and get lost in the noise. If we all had the exact same bias then it would be a huge problem.

> We’re really not that vulnerable to such things as a species, because we as individuals all have our own minds and our own sets of biases that cancel out and get lost in the noise. [Citation Needed] Just because if you have a species-wide bias, people within the species would not easily recognize it . You can't claim with a straight face that "we're really not that vulnerable to such things". For example, I think i…

> people within the species would not easily recognize it

[Citation Needed]

Sorry, but I had to. There's easy counterexamples of true, species-wide biases that we're fully aware of. Optical illusions, cognitive biases, cultural universals (community-sanctioned relationships/marriage, inheritance, ceremonial treatment of the dead). What we don't have are universal biases towards believing specific facts or stories.

Re: Where the goblins came from

#669

Post by nickpsecurity » Fri, May 01, 2026, 2:54 AM UTC

collingreen wrote: post #564

yesco wrote: post #511

Earlier quoted context omitted.

This is why I believe we should only listen to amateur opinions on everything, experts simply lack historical credibility. For example I've recently purchased a healing crystal (half off) for only $5000 dollars! It cleared up the imbalanced energies my street guru told me about right away. I would never have been made aware about the consequences of imbalanced energies in the first place if I had asked an expert inst…

Ironically the street guru hucksters might have a better track record than the dangerous products mentioned above. Less charitably, it's a mistake to imply that simply being a bigger corporation makes you go from street guru to "expert". Bigger company trying to make money off of you at any risk to you is just the same bucket at a different scale. In this context the other side is probably "expert consumer advocate"…

It can be worse in terms of justice. You might be able to charge or win in court against a street hustler. Most people can't beat a big company in court. They usually won't even try.

Re: Where the goblins came from

#670

Post by benrutter » Fri, May 01, 2026, 11:01 AM UTC

handoflixue wrote: post #411

benrutter wrote: post #384

Earlier quoted context omitted.

Not OP, but I think the argument here would be not that LLMs "are not smart" but that smart is just the wrong category of thing to describe an LLM as. A calculator can do very complex sums very quickly, but we don't tend to call it "smart" because we don't think it's operating intelligently to some internal model of the world. I think the "LLMs are AGI" crowd would say that LLMs are , but it's perfectly consistent to…

> "we don't think it's operating intelligently to some internal model of the world" Okay, but you have to actually address why you think LLMs lack an "internal model of the world" You can train one on 1930s text, and then teach it Python in-context. They've produced multiple novel mathematical proofs now; Terrance Tao is impressed with them as research assistants. You can very clearly ask them questions about the wor…

I should say that quote was referring to a calculator - I wasn't trying to stake a position on LLMs in that comment, more just pointing out that I think its consistent to think they're helpful without thinking they have AGI.

There's obviously a lot more of a case for suggesting LLMs are generally intelligent than a calculator, but for me, I think the key point is that understanding them as "next token generators" is a lot more helpful to explain things like hallucinations and some of the other issues/loops they get into.

For me, if understanding models as "generally intelligent agents operating with an internal model of the world" explained their behaviour better than "next token generators", I'd think calling them "smart" would have some justification[0]. I'm just a person on the internet though, and defining intelligence is pretty rarely clear, even without bringing LLMs into the mix.

[0] In case it's interesting to anyone, I'm basically given a half-baked version of how Daniel Dennet defined intention: https://en.wikipedia.org/wiki/Intentional_stance