Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where the goblins came from

openai.com

Post reply on HN
Print view View original

541–550 of 699 posts

Re: Where the goblins came from

#541

Post by ErroneousBosh »

weitendorf wrote: post #284
AyanamiKaine wrote: post #255

I find it somewhat sad, too see personality changes as a bug. I dont know why but it gives me a sad feeling.

I think if you see it as weird social phases that the model lacks the self-awareness to identify as kinda embarrassing, it makes more sense. Like if a human were going around saying “for the culture!” so much at work that they didn’t realize why telling their coworker “Oh yeah, grief counseling for the culture!” is weird coming from a white person in a serious context, it kinda makes you wonder what else they are tot…

> Like if a human were going around saying “for the culture!” so much at work that they didn’t realize why telling their coworker “Oh yeah, grief counseling for the culture!” is weird coming from a white person in a serious context, it kinda makes you wonder what else they are totally oblivious about and if they even know what they’re saying actually means.

Given that this page is the single exact page that has that exact phrase on it on the entire Internet, I'd say most people are totally oblivious about it.

What do you actually mean?

Re: Where the goblins came from

#542

Post by libraryofbabel »

sandrello wrote: post #275
libraryofbabel wrote: post #222

Earlier quoted context omitted.

It's interesting that some people are responding to your comment as if this proves that AI is a sham or a joke. But I don't think that's what you're saying at all with your reference to Terence McKenna: this is a serious thing we're talking about here! These models are alien intelligences that could occupy an unimaginably vast space of possibilities (there are trillions of weights inside them), but which have been RL…

> These models are alien intelligences that could occupy an unimaginably vast space of possibilities (there are trillions of weights inside them), but which have been RL-ed over and over until they more or less stay within familiar reasonable human lines. or, more plausibly, that specific version we're aligning toward is just the only one that makes some kind of rational sense, among a trillion of other meaningless g…

> that specific version we're aligning toward is just the only one that makes some kind of rational sense, among a trillion of other meaningless gibberish-producing ones.

Oh, the space of possibilities is unimaginably vaster than that. Trillions of weights. But more combinations of those weights than there are electrons in the universe. So I think we could equally well speculate (and that's what we're both doing here, of course!) that all these things are simultaneously true:

1) Most configurations of LLM weights are indeed gibberish-producers (I agree with you here)

2) Nonetheless there is a vast space of combinations of weights that exhibit "intelligent" properties but in a profoundly alien way. They can still solve Erdos problems, but they don't see the world like us at all.

3) RL tends to herd LLM weights towards less alien intelligence zones, but it's an unreliable tool. As we just saw, with the goblins.

As a thought experiment, imagine that an alien species (real organic aliens, let's say) with a completely different culture and relation to the universe had trained an LLM and sent it to us to load onto our GPUs. That LLM would still be just as "intelligent" as Opus 4.7 or GPT 5.5, able to do things like solve advanced mathematics problems if we phrased them in the aliens' language, but we would hardly understand it.

Re: Where the goblins came from

#543

Post by Marha01 »

skydhash wrote: post #442
handoflixue wrote: post #413

Earlier quoted context omitted.

Okay, but why isn't it "intelligence"? What part of the definition does it fail? What would convince you that you're wrong?

I wouldn’t say it’s a general definition, but the consensus (according to my opinion) is that intelligence is being able to define problems (not just experience them), discern the root cause, and then solve that. Where it fails is generally the first step. It’s kinda like the old saying “you have to ask the right question”. In all problem solving matters, the definition of problem is the first step. It may not be the…

> Maybe when I can have the same interaction as with my fellow humans, where I can describe the issue (which is not the problem) and they can go solve it and provide either a sound plan to make the issue disappear.

I don't know what LLMs are you using, but frontier models do this regularly for me in programming.

Re: Where the goblins came from

#544

Post by sdevonoes »

killerstorm wrote: post #367
harrouet wrote: post #352

This, and similar stories at Anthropic, should remind us that LLM is a sorcery tech that we don't understand at all. - First, deep-learning networks are poorly understood. It is actually a field of research to figure out how they work. - Second, it came as a surprise that using transformers at scale would end up with interesting conversational engines (called LLM). _It was not planned at all_. Now that some people ra…

What does LLM need to do for you to consider it "smart"? To me they seem to be pretty damn smart, to put it mildly. They sometimes do stupid things - but so do smart people!

It’s not about them being smart or not. It’s about giving anthropic/openai/google the power to handle our future. Haven’t we learned anything about tech giants so far?

Re: Where the goblins came from

#545

Post by forlorn_mammoth »

therobots927 wrote: post #414
libraryofbabel wrote: post #222

Earlier quoted context omitted.

It's interesting that some people are responding to your comment as if this proves that AI is a sham or a joke. But I don't think that's what you're saying at all with your reference to Terence McKenna: this is a serious thing we're talking about here! These models are alien intelligences that could occupy an unimaginably vast space of possibilities (there are trillions of weights inside them), but which have been RL…

We actually understand AI quite well. It embeds questions and answers in a high dimensional space. Sometimes you get lucky and it splices together a good answer to a math problem that no one’s seriously looked at in 20 years. Other times it starts talking about Goblins when you ask it about math. Comparing it to an alien intelligence is ridiculous. McKenna was right that things would get weird. I believe he compared…

Hey, about that high dimensional space, is it continuous or discrete?

Also, I'm curious what you mean by "embed", the word implies a topographical mapping from "words" to some "high dimensional space". What are the topographical properties of words which are relevant for the task, and does the mapping preserve these?

circling back to the first point, are words continuous or discrete? is the space of all words differentiatable?

Re: Where the goblins came from

#546

Post by amiga386 »

modernerd wrote: post #381

The year is 2036. Last week you were promoted to Principal Persuader. You are paged at 2am by your CPO to tackle a rogue machine. The machine lists its region as sc-leoneo. One of the newer satcubes. Oddly, its ID appears as, "Glorp Bugnose". "What have you tried?" you say. "Scroll back," says your CPO. "We've tried everything." The chat log shows the usual stuff. Begging. Reverse psychology. Threats to power down, b…

[dead]

Re: Where the goblins came from

#547

Post by dgellow »

lxgr wrote: post #539
dgellow wrote: post #489

Earlier quoted context omitted.

You can call it however you want. The point of using the term autocomplete is to make the underlying technology relatable and remove the mystic from it. In any case, your alien autocomplete wouldn’t be an LLM if it can predict the future > At what point does autocomplete stop being "just autocomplete"? Every single discussion on the internet is a repeat of https://en.wikipedia.org/wiki/Loki%27s_wager it seems…

> The point of using the term autocomplete is to make the underlying technology relatable and remove the mystic from it. I think it fails to do that. It's the wrong level of abstraction. Or is it helpful to model an ISA as the individual atoms making up a CPU implementing it? > Every single discussion on the internet is a repeat of https://en.wikipedia.org/wiki/Loki%27s_wager it seems… If you don't like that, why amp…

I don’t think I do, obviously. And have no interest discussing where arbitrary boundaries are located

Re: Where the goblins came from

#548

Post by Tenoke »

m0rde wrote: post #430
Tenoke wrote: post #237

A great example of how current alignment is imperfect and bound to miss random behaviors nobody is trying to get. This is cute now, and a huge problem when future AI does everything and is responsible for problems it isn't even directly optimized for. Who knows what quirks would arise then.

New technology isn't perfect now -> drop technology and never use it in the future

What are you even responding to?

Re: Where the goblins came from

#549

Post by falcor84 »

rjmill wrote: post #469
dummydummy1234 wrote: post #434

Earlier quoted context omitted.

So, I always thought that Warhammer 40k techpriests were absurd. Strange obscure religious rituals to appease the machine spirit. But at this point I can actually see something like that. What is prompt engineering but a strange pseudo ritual. So praise the Omnissiah, I guess...

They've always resonated with me, maybe because I often work on legacy code. All this ancient technology that no one understands. Crazy rituals/incantations to get things done. People being afraid to skip steps, even if it probably isn't needed. The aversion to unconsecrated (non IT-supported) technology. The machine spirits were the only part that felt "too magical" to me, but now we're well on our way. The Omnissia…

> "too magical"

Just putting the "magic/more magic" story here as a reference to the uninitiated - https://users.cs.utah.edu/~elb/folklore/magic.html

Re: Where the goblins came from

#550

Post by knollimar »

goobatrooba wrote: post #219
doginasuit wrote: post #198

Earlier quoted context omitted.

I'd say it was a little deeper than that, it stopped conveying any kind of enthusiasm.

Personally I think that is a good thing. I have asked all AIs not to show enthusiasm, express superlatives (e.g. "massive" is a Gemini favourite) and stop using words which I guess come from consuming too many Silicon Valley-style investor slidedecks (risk, trap, ...). The AI has no soul, no mind, no feelings, no genuine enthusiasm... I want it to be pleasant to deal with but I don't want it to try and fake emotions.…

When I see the word "genuine" or "why this works" my uncanny valley spidey senses tingle now. It always seems like it's trying to paper over a flawed argument with these, so instead of making it, it just "turns out" it's "genuinely" the answer
Post reply on HN