Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where the goblins came from

openai.com

Post reply on HN
Print view View original

471–480 of 699 posts

Re: Where the goblins came from

#471

Post by bko »

ninalanyon wrote: post #380
bko wrote: post #356

Earlier quoted context omitted.

> Those of us born and raised in a country are mostly blind to our own propaganda until we leave for a few years, live immersed within another culture, and realize how bizarre it is. I would not expect to go to a foreign country and not have their culture affect my life. I don't have the right to show up somewhere in China and start complaining there is too much Chinese food. What is a country to you? You call it "pr…

> Why wouldn't you want AI to promote your countries values? Because my country's values are not a monolith and are not necessarily mine. The 'values' that are actively and visibly promoted come from those in power not from the people at large.

Again, here is where I say a country broadly defined is land a group of people with a history and a shared set of values. Politicians or rich people can't control values. They can try to impact them. But it's out of their control as its organic.

The good news for you is that there is competition in AI models. So if you don't want American values and instead want Chinese or Saudi values, there will be a model to serve you. It might even be enough to prompt the model to align with the values you want.

I ask again, what is a country to you?

Re: Where the goblins came from

#472

Post by Orygin »

meken wrote: post #385
Orygin wrote: post #336

Earlier quoted context omitted.

It didn't seem that deep to me. They just saw an issue with Goblins, dissected the word from the model, then it appeared again in the next version without them knowing exactly how or why. Goes to show it's all vibes when making these models. The fix is literally a prompt that says not to talk about goblins...

I’m not sure how that was your takeaway..? > We retired the “Nerdy” personality in March after launching GPT‑5.4. In training, we removed the goblin-affine reward signal and filtered training data containing creature-words, making goblins less likely to over-appear or show up in inappropriate contexts. Unfortunately, GPT‑5.5 started training before we found the root cause of the goblins. The prompt is just a short te…

Then maybe stop training and make a real fix?

If you need to put baby guardrails on your model because the training is effed up, maybe you should rethink how you make these models and how much control you really have on it.

Re: Where the goblins came from

#473

Post by hansmayer »

killerstorm wrote: post #367
harrouet wrote: post #352

This, and similar stories at Anthropic, should remind us that LLM is a sorcery tech that we don't understand at all. - First, deep-learning networks are poorly understood. It is actually a field of research to figure out how they work. - Second, it came as a surprise that using transformers at scale would end up with interesting conversational engines (called LLM). _It was not planned at all_. Now that some people ra…

What does LLM need to do for you to consider it "smart"? To me they seem to be pretty damn smart, to put it mildly. They sometimes do stupid things - but so do smart people!

How about writing "all code" this June, as Dario Amodei announced in January this year?

Re: Where the goblins came from

#474

Post by steveBK123 »

somewhatgoated wrote: post #437
steveBK123 wrote: post #423

Earlier quoted context omitted.

Humans can be governed by rules with consequences and replaced with individuals with a appropriate level of risk taking / rule following for the role.

Rules and consequences seem to apply to humans in a similar way as prompts and harnesses govern LLMs. The greater the level of power a human possesses the less they are governed by these restraints, this doesnt apply to LLMs so at least in that aspect they are an improvement. But yea we can’t really punish or inflict pain on them - this seems like a problem

I think a simpler model is variety.

There are billions of people, you can interview/hire/fire until you get the right match.

There are 2? frontier LLM providers. 5? if you are more generous / ok with more trailing edge.

Everyone thought OpenAI was great, until Claude got better in Q1 and they switched to Anthropic, and then Codex got better and a good chunk moved back to OpenAI.. Seems kind of binary currently.

Re: Where the goblins came from

#475

Post by ashtonshears »

wanderingmind wrote: post #362
harrouet wrote: post #352

This, and similar stories at Anthropic, should remind us that LLM is a sorcery tech that we don't understand at all. - First, deep-learning networks are poorly understood. It is actually a field of research to figure out how they work. - Second, it came as a surprise that using transformers at scale would end up with interesting conversational engines (called LLM). _It was not planned at all_. Now that some people ra…

Humanity has been using steel for over a millenia, however it's only in the past 100 years or so we have a good understanding of how carbon interacts with iron at an atomic level to create the strength characteristics that makes it useful. Based on this argument, we should not have used steel, until we had a complete first principles understanding.

Poor correlation comparing physical material to computer technology

Re: Where the goblins came from

#476

Post by steveBK123 »

someguyiguess wrote: post #453
steveBK123 wrote: post #423

Earlier quoted context omitted.

Humans can be governed by rules with consequences and replaced with individuals with a appropriate level of risk taking / rule following for the role.

You clearly have never met a human

If you cannot get humans to do roughly what you want as a manager, good luck with LLMs.

Re: Where the goblins came from

#477

Post by steveBK123 »

handoflixue wrote: post #438
steveBK123 wrote: post #423

Earlier quoted context omitted.

Humans can be governed by rules with consequences and replaced with individuals with a appropriate level of risk taking / rule following for the role.

That seems like it applies just fine to LLMs as well: You can replace an LLM with a different model, different prompts, etc. for the appropriate level of risk taking. Rule following is even easier, given you can sandbox them.

Theres at best a handful of frontier models vs billions of people and millions of SWEs.

Re: Where the goblins came from

#478

Post by irishcoffee »

i_have_an_idea wrote: post #463
wanderingmind wrote: post #362

Earlier quoted context omitted.

Humanity has been using steel for over a millenia, however it's only in the past 100 years or so we have a good understanding of how carbon interacts with iron at an atomic level to create the strength characteristics that makes it useful. Based on this argument, we should not have used steel, until we had a complete first principles understanding.

What if you substituted "steel" with "asbestos" in your argument.

Asbestos, lead paint, cigarettes, heroin(perscribed generously for basically whatever the doc felt like), "Radithor" (patent medicine containing radium-226 and 228, marketed as a "perpetual sunshine" energy tonic and cure for over 150 diseases), bloodletting, mercury treatments for syphilis, tobacco smoke enemas (yep that was a real thing), milk-based blood transfusions.

Didn't understand those either and used the fuck out of them because "the experts" said we should.

Re: Where the goblins came from

#479

Post by djeastm »

goobatrooba wrote: post #223
christoph wrote: post #82

Earlier quoted context omitted.

Does nobody else laugh that a company supposedly worth more than almost anything else at the moment, is basically hacking around a load of text files telling their trillion dollar wonder machine it absolutely must stop talking to customers about goblins, gremlins and ogres? The number one discussion point, on the number one tech discussion site. This literally is, today, the state of the art. McKenna looks more corre…

Indeed. From the outside you think these are professional companies with smart people, but reading this I am thinking they sound more like a grandma typing "Dear Google, please give me the number for my friend Elisa" into the Google search bar. Basically, they don't seem to understand their own product.. they have learned how to make it behave in certain way but they don't truly understand how it works or reaches it'…

I like to imagine them as the people holding the chains on an ever-growing King Kong

Re: Where the goblins came from

#480

Post by dreis_sw »

jagged-chisel wrote: post #387
lukan wrote: post #375

Earlier quoted context omitted.

Well, we did build airplanes out of steel, but there are better (lighter) materials avaiable. But the developement of car engines did directly enabled airplane engines. Not sure if this is the right analogy path, but I kind of suspect similar with LLM's/transformers. They will be a important part.

An important stepping stone, perhaps. But I don’t think the final AGI thing will necessarily contain LLMs.

History shows continuous evolution, there won't be a "final AGI thing". The definition of AGI is so vague anyways that any conversation around it is hardly useful. 5 years ago, what we have today would have been considered AGI.
Post reply on HN