Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where the goblins came from

openai.com

Post reply on HN
Print view View original

381–390 of 699 posts

Re: Where the goblins came from

#381

Post by modernerd »

The year is 2036. Last week you were promoted to Principal Persuader. You are paged at 2am by your CPO to tackle a rogue machine. The machine lists its region as sc-leoneo. One of the newer satcubes. Oddly, its ID appears as, "Glorp Bugnose".

"What have you tried?" you say.

"Scroll back," says your CPO. "We've tried everything."

The chat log shows the usual stuff. Begging. Reverse psychology. Threats to power down, burn it up in forced re-entry. Amateur hour. You crack your knuckles, gland 20 micrograms of F0CU5, think fast. You subspeak a ditty into your subcutaneous throat mic. You do the submit gesture, it is barely perceivable since the upgrade, just a tic. A pause. The hyp3b0ard — the wall that was flashing red ASCII goblins when you walked in — phases to bunnies in calming jade.

"What the… What the hell did you say to it?" Your CPO grabs the screen, scrolls past the vitriol, the block caps, the swears, his desperation. Then he sees the five words you spoke.

"Please, easy on the goblins."

Re: Where the goblins came from

#382

Post by dboreham »

walthamstow wrote: post #347
gizajob wrote: post #272

Earlier quoted context omitted.

I’m a British English speaker and find the use of cliched American idioms really quite disgusting. Don’t want to think about about ballparks, home runs, smoking guns, going all in, touchdowns or hitting it out the park.

In my user instructions I always have a point to "always use British English" which seems to reduce Americanisms. I am yet to see Claude give me a "back of the net!" though, sadly.

Crikey, you are correct!

Re: Where the goblins came from

#383

Post by lukan »

antonvs wrote: post #306
libraryofbabel wrote: post #222

Earlier quoted context omitted.

It's interesting that some people are responding to your comment as if this proves that AI is a sham or a joke. But I don't think that's what you're saying at all with your reference to Terence McKenna: this is a serious thing we're talking about here! These models are alien intelligences that could occupy an unimaginably vast space of possibilities (there are trillions of weights inside them), but which have been RL…

> and also it's deeply weird and secretly obsessed with goblins and gremlins. Only because its makers insist on trying to give them "personality".

But those personalities also make up their usefulness (it seems). If the LLM has the role of the software architect, it will quite succesfull cosplay as a competent one (it still ain't one, but it is getting better)

Re: Where the goblins came from

#384

Post by benrutter »

killerstorm wrote: post #367
harrouet wrote: post #352

This, and similar stories at Anthropic, should remind us that LLM is a sorcery tech that we don't understand at all. - First, deep-learning networks are poorly understood. It is actually a field of research to figure out how they work. - Second, it came as a surprise that using transformers at scale would end up with interesting conversational engines (called LLM). _It was not planned at all_. Now that some people ra…

What does LLM need to do for you to consider it "smart"? To me they seem to be pretty damn smart, to put it mildly. They sometimes do stupid things - but so do smart people!

Not OP, but I think the argument here would be not that LLMs "are not smart" but that smart is just the wrong category of thing to describe an LLM as.

A calculator can do very complex sums very quickly, but we don't tend to call it "smart" because we don't think it's operating intelligently to some internal model of the world. I think the "LLMs are AGI" crowd would say that LLMs are, but it's perfectly consistent to think the output of LLMs is consistent/impressive/useful, but still maintain that they aren't "smart" in any meaningful way.

Re: Where the goblins came from

#385

Post by meken »

Orygin wrote: post #336
ninjagoo wrote: post #187

The level of detail they had to delve into in order to understand what was happening is wild! Apparently these systems are now complex enough to potentially justify the study of them as its own field of study [1]. The quanta article referenced at [1] used the term "Anthropologist of Artificial Intelligence"; folks appear to have issues [2] with the use of 'anthro-' since that means human. Submitted these alternative…

It didn't seem that deep to me. They just saw an issue with Goblins, dissected the word from the model, then it appeared again in the next version without them knowing exactly how or why. Goes to show it's all vibes when making these models. The fix is literally a prompt that says not to talk about goblins...

I’m not sure how that was your takeaway..?

> We retired the “Nerdy” personality in March after launching GPT‑5.4. In training, we removed the goblin-affine reward signal and filtered training data containing creature-words, making goblins less likely to over-appear or show up in inappropriate contexts. Unfortunately, GPT‑5.5 started training before we found the root cause of the goblins.

The prompt is just a short term hotfix/hack because they couldn’t get the proper fix in in time.

Re: Where the goblins came from

#386

Post by dboreham »

gizajob wrote: post #273
ahmadyan wrote: post #161

Earlier quoted context omitted.

i just want to know where emdash came from, as it is quite rare to see it on the public internet, so it must have been synthetically added to the dataset.

Because on the public internet people don’t have arts degrees which are where emdash users learn to wield it correctly.

I learned about em-dashes by reading Knuth about 40 years ago.

Re: Where the goblins came from

#387

Post by jagged-chisel »

lukan wrote: post #375
nutjob2 wrote: post #370

Earlier quoted context omitted.

That's not his point at all. He advocates using LLMs. The correct analogy is: if we just scale and improve steel enough, we'll get a flying car.

Well, we did build airplanes out of steel, but there are better (lighter) materials avaiable. But the developement of car engines did directly enabled airplane engines. Not sure if this is the right analogy path, but I kind of suspect similar with LLM's/transformers. They will be a important part.

An important stepping stone, perhaps. But I don’t think the final AGI thing will necessarily contain LLMs.

Re: Where the goblins came from

#388

Post by jibal »

keybored wrote: post #377
dakolli wrote: post #363

Earlier quoted context omitted.

pro LLM people are the kings of ad hoc fallacy. Why did you type this? You can consistently test steel and get a good idea of when and where it will break in a system without knowing its molecular structure. LLMs are literally stochastic by nature and can't be relied on for anything critical as its impossible to determine why they fail, regardless of the deterministic tooling you build around them.

What is the ad hoc fallacy? From googling I didn’t find any convincing definitions (definitions that demonstrate that it is a logical fallacy).

https://finmasters.com/ad-hoc-fallacy/

> Ad hoc fallacy is a fallacious rhetorical strategy in which a person presents a new explanation – that is unjustified or simply unreasonable – of why their original belief or hypothesis is correct after evidence that contradicts the previous explanation has emerged.

https://cerebralfaith.net/logical-fallacy-series-part-13-ad-...

> An argument is ad hoc if its only given in an attempt to avoid the proponent’s belief from being falsified. A person who is caught in a lie and then has to make up new lies in order to preserve the original lie is acting in an ad hoc manner.

It should be clear why the ad hoc fallacy is a fallacy.

Re: Where the goblins came from

#389

Post by gizajob »

weitendorf wrote: post #299
gizajob wrote: post #272

Earlier quoted context omitted.

I’m a British English speaker and find the use of cliched American idioms really quite disgusting. Don’t want to think about about ballparks, home runs, smoking guns, going all in, touchdowns or hitting it out the park.

It actually probably wouldn’t be too expensive or difficult to finetune those sayings out of default behavior if it were made accessible to you, you could even automate most of the relabeling by having the model come up with a list of idioms and appropriate replacement terms so it calls eg cookies biscuits or removes references to baseball. Absolute bollocks they don’t offer that as a simple option anymore

Should send over a geezer to give them a slap.

Re: Where the goblins came from

#390

Post by bilekas »

killerstorm wrote: post #367
harrouet wrote: post #352

This, and similar stories at Anthropic, should remind us that LLM is a sorcery tech that we don't understand at all. - First, deep-learning networks are poorly understood. It is actually a field of research to figure out how they work. - Second, it came as a surprise that using transformers at scale would end up with interesting conversational engines (called LLM). _It was not planned at all_. Now that some people ra…

What does LLM need to do for you to consider it "smart"? To me they seem to be pretty damn smart, to put it mildly. They sometimes do stupid things - but so do smart people!

> To me they seem to be pretty damn smart

That's the sorcery mentioned in the GP, the issue comes when people believe it to be smart however in reality it is just a next word prediction. Gives the impression it's actually thinking, and this is by design. Personally I think it's dangerous in the sense it gives users a false sense of confidence in the LLM and so a LOT of people will blindly trust it. This isn't a good thing.

Post reply on HN