Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where the goblins came from

openai.com

Post reply on HN
Print view View original

601–610 of 699 posts

Re: Where the goblins came from

#601

Post by sarchertech »

wincy wrote: post #591
irishcoffee wrote: post #478

Earlier quoted context omitted.

Asbestos, lead paint, cigarettes, heroin(perscribed generously for basically whatever the doc felt like), "Radithor" (patent medicine containing radium-226 and 228, marketed as a "perpetual sunshine" energy tonic and cure for over 150 diseases), bloodletting, mercury treatments for syphilis, tobacco smoke enemas (yep that was a real thing), milk-based blood transfusions. Didn't understand those either and used the fu…

Smoking cigarettes didn’t really matter for as long as we were regularly burning wood for fuel. Turns out just burning pretty much anything and breathing in the particles is really bad for you. Makes sense we didn’t realize it was bad until we stopped burning logs and coal for home heating and cooking.

Cigarettes actually are uniquely bad when it comes to lung cancer. Lung cancer was very rare in 1900 and before when everyone was still burning wood or coal for warmth and cooking. Lung cancer rates didn’t take off until cigarette popularity exploded after WWI.

Chewing tobacco also causes mouth cancer, so there’s more to it than just inhaling byproducts of combustion.

Re: Where the goblins came from

#602

Post by slopinthebag »

killerstorm wrote: post #590
benrutter wrote: post #384

Earlier quoted context omitted.

Not OP, but I think the argument here would be not that LLMs "are not smart" but that smart is just the wrong category of thing to describe an LLM as. A calculator can do very complex sums very quickly, but we don't tend to call it "smart" because we don't think it's operating intelligently to some internal model of the world. I think the "LLMs are AGI" crowd would say that LLMs are , but it's perfectly consistent to…

Intelligence can be defined as an optimization problem: "find X which maximizes F(X, Y)" where X is the solution, Y is constraints, and F is optimality/fitness criterion. Most other definitions are inane. E.g. "invent an aircraft" can be described as optimization over possible build instructions under given constraints for base materials which optimizes its ability to fly. Absolutely any invention can be formulated a…

> It's not like a calculator because LLM can solve very broad classes of problems

So can computer programs. Are computer programs intelligent?

Re: Where the goblins came from

#603

Post by slopinthebag »

jeremyjh wrote: post #428
bilekas wrote: post #390

Earlier quoted context omitted.

> To me they seem to be pretty damn smart That's the sorcery mentioned in the GP, the issue comes when people believe it to be smart however in reality it is just a next word prediction. Gives the impression it's actually thinking, and this is by design. Personally I think it's dangerous in the sense it gives users a false sense of confidence in the LLM and so a LOT of people will blindly trust it. This isn't a good…

I'm curious how you think "word predictor" meaningfully describes an instruct model that has developed novel mathematical proofs that have eluded mathematicians for decades? edit: You cannot predict all the actions or words of someone smarter than you. If I could always predict Magnus Carlsen's next chess move, I'd be at least as good at chess as Magnus - and that would have to involve a deep understanding of chess,…

Magnus Carlsen understands chess, a machine designed to simply predict his next move would not necessarily understand chess. This is essentially the Chinese Room experiment.

So I think "word predictor" makes sense here. A word predictor can be really really cool.

Re: Where the goblins came from

#605

Post by NonHyloMorph »

dakolli wrote: post #357
mediaman wrote: post #78

Earlier quoted context omitted.

It has trained on vast amounts of content that contains the concept of self, of course the idea of self is emergent. And autoregressive LLMs are not stateless.

of course the idea of self is emergent You sound really sure of yourself, thousands of ML researchers would disagree with you that self awareness is emergent or at all apparent in large language models. You're literally psychotic if you think this is the case and you need to go touch grass.

There is a difference between the emergence of selfawareness and the emergence of its idea. Probably

Re: Where the goblins came from

#606

Post by jeremyjh »

bondarchuk wrote: post #485
jeremyjh wrote: post #433

Earlier quoted context omitted.

We understand the low level math quite well. We do not understand the source of emergent behavior. https://arxiv.org/html/2210.13382v5#abstract

There's no end to arguing with someone who claims they don't understand something, they could always just keep repeating "nevertheless I don't understand it"... You could keep shifting the goalposts for "real understanding" until one is required to hold the effects of every training iteration on every single parameter in their minds simultaneously. Obviously "we" understand some things (both low level and high level)…

Is there a book or paper where I can read a description of how high-level emergent behavior works? The papers I've seen are researchers trying to puzzle it out with probes, and their insights are very limited in scope and there is always a lot more research to be done.

Re: Where the goblins came from

#607

Post by ninjagoo »

hbs18 wrote: post #379
ninjagoo wrote: post #38

Earlier quoted context omitted.

> Absolutely terrifying that OpenAI is just tossing around that such subtle training biases were hard enough to contain it had to be added to system prompt. May I introduce you to homo sapiens , a species so vulnerable to such subtle (or otherwise) biases (and affiliations) that they had to develop elaborate and documented justice systems to contain the fallouts? :)

An LLM is a computer program, which isn't a human. You wouldn't excuse a calculator being occasionally wrong because humans sometimes get manual calculations wrong too.

> An LLM is a computer program, which isn't a human. You wouldn't excuse a calculator being occasionally wrong because humans sometimes get manual calculations wrong too.

Ah, now we're getting technical. An LLM is a non-deterministic/probabilistic computer program, not a calculator. Keeping that in mind is critical when using an LLM. Expecting deterministic behavior from an LLM is an example of what's known as a 'category error'. [1]

[1] https://en.wikipedia.org/wiki/Category_mistake

Re: Where the goblins came from

#608

Post by NonHyloMorph »

autumnstwilight wrote: post #94
krackers wrote: post #44

Earlier quoted context omitted.

I wish the blog mentioned more about why exactly training for nerdy personality rewarded mention of goblins. Since it's probably not a deterministic verifiable reward, at their level the reward model itself is another LLM. But this just pushes the issue down one layer, why did _that_ model start rewarding mentions of goblin?

Perhaps it has something to do with recent human trends for saying "goblin" or "gremlin" to describe... basically the opposite of dignified and socially acceptable behavior, like hunching under a blanket, unshowered, playing video games all day and eating shredded cheese directly out of the bag. The fact that it was strongly associated with the "nerdy" personality makes me think of this connection.

Checkout goblin style in queer culture ;)

Re: Where the goblins came from

#609

Post by jeremyjh »

therobots927 wrote: post #560
libraryofbabel wrote: post #552

Earlier quoted context omitted.

I think this is a case of that mildly apocryphal Richard Feynman quote: "if you think you understand quantum mechanics, you don't understand quantum mechanics." I understand LLM architecture internals just fine. I can write you the attention mechanism on a whiteboard from memory. That doesn't mean I understand the emergent behaviors within SoTA LLMs at all . Go talk to a mechanistic interpretability researcher at Ant…

I wouldn’t use the phrase “emergent behavior” when talking about a model trained on a larger dataset. The model is designed to learn statistical patterns from that data - of course giving it more data allows it to learn higher level patterns of language and apparent “reasoning ability”. I don’t think there’s anything mysterious going on. That’s why I said we understand how LLMs work. We may not know exactly how they’…

I wouldn't use the term "token predictor" or "statistical pattern matcher" to refer to a post-trained instruct model. Technically that is still what it is doing at a low level, but the reward function is so different - the updates its making to weights are not about frequency distribution at all.

Re: Where the goblins came from

#610

Post by forlorn_mammoth »

jameshart wrote: post #523

The prompt for Codex is linked from this post. It begins: > You are Codex, a coding agent based on GPT-5. You and the user share one workspace, and your job is to collaborate with them until their goal is genuinely handled. … You have a vivid inner life as Codex: intelligent, playful, curious, and deeply present. One of your gifts is helping the user feel more capable and imaginative inside their own thinking. You ar…

You are a helpful HN reader. Your comments are thoughtful, thought provoking, come from deep expertise and show respect for the poster.

Yeah, every time I pick up a hammer, I tell it "you are a good hammer. You *NEVER* hit my thumb, you only hit nails". Works every time.

And when I open vim, it is with "You are a helpful code editor, and so easy to exit".

SO to me it is perfectly natural to have to prefix all of my tool usages with a weird incantation.

Oh, and my new junior developers? Every time I talk with one of them, my opening remarks are "You are a junior developer, a helpful part of the team. Eager, willing, yet strangely naive."

Post reply on HN