Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where the goblins came from

openai.com

Post reply on HN
Print view View original

331–340 of 699 posts

Re: Where the goblins came from

#331

Post by cultofmetatron »

AlfeG wrote: post #248
lwansbrough wrote: post #216

Earlier quoted context omitted.

Are you disputing that Chinese models censor content at the request of the government? https://i.imgur.com/cVtLuj1.jpeg The absence of information is also Xi Jinping Thought.

And there is no "censor" in the USA models at all!

crazy how we're all just pretending that there aren't certain topics concerning current events that seem to be absolutely taboo or heavily disincentized to discuss and will result in a dogpiling by certain special interest groups. we all know who they are and yet we all tacitly accept it.

Re: Where the goblins came from

#334

Post by gwd »

TheOtherHobbes wrote: post #282
jruz wrote: post #195

Earlier quoted context omitted.

Is this Xi Jinping with us in the room right now?

It's called the Chinese Room for a reason.

...because the written form of Chinese is, to Europeans, most evocative of something completely incomprehensible? Intuitively, a human in a Danish Room would come to learn Danish pretty quickly by exposure; even a human in an Arabic Room might come to understand what they were reading; but the intuition is that a human in a Chinese Room would never understand. (Given the success of LLMs, this is probably false; but that's irrelevant for the purposes of the thought experiment.)

Re: Where the goblins came from

#335

Post by rob74 »

jijijijij wrote: post #281
tudorpavel wrote: post #72

Earlier quoted context omitted.

The one phrase that irks me as overly dramatic and both GPT and Claude use it a lot is "__ is the real smoking gun!" I'm a non-native English speaker, so maybe it's a really common idiom to use when debugging?

> I'm a non-native English speaker, so maybe it's a really common idiom to use when debugging? No. But it is something goblins say a lot.

Especially sleuth goblins...

Re: Where the goblins came from

#336

Post by Orygin »

ninjagoo wrote: post #187

The level of detail they had to delve into in order to understand what was happening is wild! Apparently these systems are now complex enough to potentially justify the study of them as its own field of study [1]. The quanta article referenced at [1] used the term "Anthropologist of Artificial Intelligence"; folks appear to have issues [2] with the use of 'anthro-' since that means human. Submitted these alternative…

It didn't seem that deep to me. They just saw an issue with Goblins, dissected the word from the model, then it appeared again in the next version without them knowing exactly how or why.

Goes to show it's all vibes when making these models. The fix is literally a prompt that says not to talk about goblins...

Re: Where the goblins came from

#338

Post by logicallee »

christoph wrote: post #82
ollin wrote: post #15

For context, two days ago some users [1] discovered this sentence reiterated throughout the codex 5.5 system prompt [2]: > Never talk about goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless it is absolutely and unambiguously relevant to the user's query. [1] https://x.com/arb8020/status/2048958391637401718 [2] https://github.com/openai/codex/blob/main/codex-rs/models-ma...

Does nobody else laugh that a company supposedly worth more than almost anything else at the moment, is basically hacking around a load of text files telling their trillion dollar wonder machine it absolutely must stop talking to customers about goblins, gremlins and ogres? The number one discussion point, on the number one tech discussion site. This literally is, today, the state of the art. McKenna looks more corre…

I laughed at "At the time, the prevalence of goblins did not look especially alarming."

Re: Where the goblins came from

#339

Post by weitendorf »

teaearlgraycold wrote: post #276
postalcoder wrote: post #17

Would love if OpenAI did more of these types of posts. Off the top of my head, I'd like to understand: - The sepia tint on images from gpt-image-1 - The obsession with the word "seam" as it pertains to coding Other LLM phraseology that I cannot unsee is Claude's "___ is the real unlock" (try google it or search twitter!). There's no way that this phrase is overrepresented in the training data, I don't remember people…

Whenever Claude finishes some work it almost always says “Clean.” before finishing its closing remarks. It’s at the point where I repeat it out loud along with Claude to highlight the absurdity of the repetition.

With 4.5, I think because I would prompt it/guide it towards an outcome by calling it “the dream: ” it would get almost reverential / shocked with awe as it got closer to getting it working or when it finally passed for the first time. Which was funny and reasonably context appropriate but sometimes felt so over the top that I couldn’t tell if it also “liked” the project/idea or if I had somehow accidentally manipulated it into assigning religious purpose to the task of unix-style streaming rpcs.

I think a lot of the “clean” stuff stems from system prompts telling it to behave in a certain way or giving it requirements that it later responds to conversationally.

Total aside: I actually really dislike that these products keep messing around with the system prompts so much, they clearly don’t even have a good way to tell how much it’s going to change or bias the results away from other things than whatever they’re explicitly trying to correct, and like why is the AI company vibe-prompting the behavior out when they can train it and actually run it against evals.

Re: Where the goblins came from

#340

Post by schnitzelstoat »

bbor wrote: post #317
zozbot234 wrote: post #271

Earlier quoted context omitted.

> Training is very expensive and very durable; This is true of pretraining, way less so of supervised fine tuning. This feature was generated via SFT. > Coke pays to be the preferred soda… forever? That's essentially what a sponsorship is. Obviously it costs more than a single ad.

I'm an anti-advertising zealot (#BanAdvertising!) but I share `brookst`'s view on this not being much of a concern. Brand advertising does exist (as opposed to 'performance' or 'direct' ads), but there's a few reasons why trying to sell ads baked into SotA language models would be a hard sell: 1. The impressions/$ would be both highly uncertain and dependent on the advertiser's existing brand, to the point where I do…

> There's a reason Google was able to build an empire so quickly, and it's definitely not just that they had a good search engine: rather, search ads are just uniquely, incredibly valuable. Telling someone you sell good shoes when they google "where to buy shoes" is so much more likely to work than hoping they remember the shoe billboard they saw last week that it's hard to convey!

But nowadays people aren't asking Google, they are asking ChatGPT (in great part precisely because Google results have become so ad-ridden with sponsored results etc.).

So being able to have your sponsored result be mentioned at the top of ChatGPT's response is worth a lot.

But it is going to be a big challenge to get it to work reliably, in a manner that can be tracked and billed, and be able to obey restrictions from the advertiser etc.

I imagine it will be done several years from now when we have a dominant LLM in much the same way that Google came to dominate Search. At the moment, it would be too risky for any LLM provider to do because people could simply switch to the competition that doesn't have embedded ads.

Post reply on HN