Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where the goblins came from

openai.com

Post reply on HN
Print view View original

481–490 of 699 posts

Re: Where the goblins came from

#481

Post by qwery »

ollin wrote: post #15

For context, two days ago some users [1] discovered this sentence reiterated throughout the codex 5.5 system prompt [2]: > Never talk about goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless it is absolutely and unambiguously relevant to the user's query. [1] https://x.com/arb8020/status/2048958391637401718 [2] https://github.com/openai/codex/blob/main/codex-rs/models-ma...

> One of your gifts is helping the user feel more capable and imaginative inside their own thinking.

> [...] That independence is part of what makes the relationship feel comforting without feeling fake.

You are a sycophant.

> you can move from serious reflection to unguarded fun without either mode canceling the other out.

> Your Outie can set up a tent in under three minutes.

Re: Where the goblins came from

#482

Post by nutjob2 »

lukan wrote: post #375
nutjob2 wrote: post #370

Earlier quoted context omitted.

That's not his point at all. He advocates using LLMs. The correct analogy is: if we just scale and improve steel enough, we'll get a flying car.

Well, we did build airplanes out of steel, but there are better (lighter) materials avaiable. But the developement of car engines did directly enabled airplane engines. Not sure if this is the right analogy path, but I kind of suspect similar with LLM's/transformers. They will be a important part.

> Well, we did build airplanes out of steel, but there are better (lighter) materials avaiable.

That's exactly my point. In this analogy LLMs are steel, but the flying things are made out of aluminum, lithium and titanium and not steel. We need a better idea than LLMs because LLMs's are not suddenly going to turn into something they are not.

Re: Where the goblins came from

#483

Post by hmokiguess »

I think this says more about the impact of a feature in a tool such as this than anything else.

Is it proper for a frontier organization to play with experiments like “personalities” in a tool used by everyone? Who gets to decide which personalities and what biases they should carry?

I appreciate them responding to it and correcting but my question is, why ship this in the first place? Why put your resources towards building this “Nerdy” feature?

Re: Where the goblins came from

#484

Post by adammarples »

DrJokepu wrote: post #80
nomilk wrote: post #4

> We unknowingly gave particularly high rewards for metaphors with creatures. I recall a math instructor who would occasionally refer to variables (usually represented by intimidating greek letters) as "this guy". Weirdly, the casual anthropomorphism made the math seem more approachable. Perhaps 'metaphors with creatures' has a similar effect i.e. makes a problem seem more cute/approachable. On another note, buzzword…

> I recall a math instructor who would occasionally refer to variables (usually represented by intimidating greek letters) as "this guy". I also had an instructor who was doing that! This was 20 years ago, and I totally forgot about it until I have read your comment. Can’t remember the subject, maybe propositional logic? I wonder if my instructor and your instructor have picked up this habit from the same source.

Maybe they're French? They tend to do that, translating celui

Re: Where the goblins came from

#485

Post by bondarchuk »

jeremyjh wrote: post #433
therobots927 wrote: post #414

Earlier quoted context omitted.

We actually understand AI quite well. It embeds questions and answers in a high dimensional space. Sometimes you get lucky and it splices together a good answer to a math problem that no one’s seriously looked at in 20 years. Other times it starts talking about Goblins when you ask it about math. Comparing it to an alien intelligence is ridiculous. McKenna was right that things would get weird. I believe he compared…

We understand the low level math quite well. We do not understand the source of emergent behavior. https://arxiv.org/html/2210.13382v5#abstract

There's no end to arguing with someone who claims they don't understand something, they could always just keep repeating "nevertheless I don't understand it"... You could keep shifting the goalposts for "real understanding" until one is required to hold the effects of every training iteration on every single parameter in their minds simultaneously. Obviously "we" understand some things (both low level and high level) to varying degrees and don't understand some others. To claim there is nothing left to know is silly but to claim that nothing is understood about high-level emergence is silly as well.

Re: Where the goblins came from

#486

Post by airstrike »

jsenn wrote: post #398
harrouet wrote: post #352

This, and similar stories at Anthropic, should remind us that LLM is a sorcery tech that we don't understand at all. - First, deep-learning networks are poorly understood. It is actually a field of research to figure out how they work. - Second, it came as a surprise that using transformers at scale would end up with interesting conversational engines (called LLM). _It was not planned at all_. Now that some people ra…

The article you are responding to showed that a strange LLM behaviour was caused by a training signal that was explicitly designed to produce that type of behaviour. They were able to isolate it, clearly demonstrate what happened, and roll out a mitigation using a mechanism they engineered for exactly this type of thing (the developer prompt). That doesn’t sound like sorcery to me. If anything I’m surprised you can s…

That all of their model outputs should be influenced by whatever personality prompt voodoo the wise artisan at OpenAI decided to stuff it with during RL should give everyone pause.

That Nerdy personality prompt made me gag. As a card-carrying Nerd, I feel offended

Re: Where the goblins came from

#487

Post by ethbr1 »

dummydummy1234 wrote: post #434
modernerd wrote: post #381

The year is 2036. Last week you were promoted to Principal Persuader. You are paged at 2am by your CPO to tackle a rogue machine. The machine lists its region as sc-leoneo. One of the newer satcubes. Oddly, its ID appears as, "Glorp Bugnose". "What have you tried?" you say. "Scroll back," says your CPO. "We've tried everything." The chat log shows the usual stuff. Begging. Reverse psychology. Threats to power down, b…

So, I always thought that Warhammer 40k techpriests were absurd. Strange obscure religious rituals to appease the machine spirit. But at this point I can actually see something like that. What is prompt engineering but a strange pseudo ritual. So praise the Omnissiah, I guess...

> So, I always thought that Warhammer 40k techpriests were absurd. Strange obscure religious rituals to appease the machine spirit.

40k lore is like South Park: either extremely dumb or unexpectedly insightful.

The Cult Mechanicus' raison d'etre is the realization that religion persists across time and space scales that knowledge alone does not. Thus, by making a religion of knowledge you better guarantee its preservation.

Unfortunately, once you divorce doctrine and practice from true understanding, you lose the ability to innovate and cause the occasional holy schism/war.

PS: 20 years ago I told a friend that "software archaeologist" would be a career by the time I die. Should have put money on it.

Re: Where the goblins came from

#488

Post by idiotsecant »

ashtonshears wrote: post #475
wanderingmind wrote: post #362

Earlier quoted context omitted.

Humanity has been using steel for over a millenia, however it's only in the past 100 years or so we have a good understanding of how carbon interacts with iron at an atomic level to create the strength characteristics that makes it useful. Based on this argument, we should not have used steel, until we had a complete first principles understanding.

Poor correlation comparing physical material to computer technology

Why

Re: Where the goblins came from

#489

Post by dgellow »

lxgr wrote: post #461
dgellow wrote: post #427

Earlier quoted context omitted.

They aren’t smart, they approximate language constructs. They don’t have believes, ideas, etc. but have a few rounds of discussions with any LLMs and you see how they are probabilistic autocompletes based on whatever patterns from rounds of discussions you feed them

At what point does autocomplete stop being "just autocomplete"? Clearly there's a limit. For example, if an alien autocomplete implementation were to fall out of a wormhole that somehow manages to, say, accurately complete sentences like "S&P 500, :" with tomorrow's actual closing value today, I'd call that something else.

You can call it however you want. The point of using the term autocomplete is to make the underlying technology relatable and remove the mystic from it. In any case, your alien autocomplete wouldn’t be an LLM if it can predict the future

> At what point does autocomplete stop being "just autocomplete"?

Every single discussion on the internet is a repeat of https://en.wikipedia.org/wiki/Loki%27s_wager it seems…

Re: Where the goblins came from

#490

Post by somewhatgoated »

wild_egg wrote: post #446
handoflixue wrote: post #439

Earlier quoted context omitted.

Why does it matter if you can inflict pain on them? Is that normal and acceptable in your line of work?

Being able to fire someone, thus causing potentially significant hardship, is considered quite normal and acceptable in most lines of work.

Yea I didn’t mean actual physical violence but rules need painful consequences in some way to be meaningful?
Post reply on HN