Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where the goblins came from

openai.com

Post reply on HN
Print view View original

531–540 of 699 posts

Re: Where the goblins came from

#531

Post by aldebaran1 »

surgical_fire wrote: post #527
aldebaran1 wrote: post #520

Earlier quoted context omitted.

[dead]

No. > When some normally ductile metal alloys are cooled to relatively low temperatures, they become susceptible to brittle fracture—that is, they experience a ductile-to-brittle transition upon cooling through a critical range of temperatures. That we did not know how steel behaved under low temperatures in building ship husks does not make it unpredictable. It was an engineering failure. Unpredictability would be i…

>That we did not know how steel behaved under low temperatures in building ship husks does not make it unpredictable.

Yes it does. Or rather, 'steel as used in shipbuilding' is unpredictable (a pedantic distinction). If the properties of steel were fully understood then someone would have identified the brittle fracture concern. They did not, hence the steel-ship system behavior was not predicted. Whether it was /predictable/ is a exercise in hindsight.

>Unpredictability would be if steel behaved fine in 2 ships, cracked in 3 ships under low pressure for becoming brittle, in another ship it turned into gelatine, and in another it behaved fine but gained a pink color.

That's not how LLMs work either. If you could control all the parameters that go into training and using an LLM, they would be predictable in the same sense (in theory, given enough time to analyze inputs/outputs given fixed process parameters).

Also steel does in fact behave probabilistically, for example in the distribution of assumed pre-existing flaw sizes in castings which are very important for the structural performance. Not all liberty ships cracked.

Re: Where the goblins came from

#532

Post by surgical_fire »

carlosjobim wrote: post #526
surgical_fire wrote: post #502

Earlier quoted context omitted.

No, it was actually engines. The mechanism behind engines were fully understood, any experiments with engines were reproducible and measurable. You could get an engine and create schematics by reverse engireening it. LLMs, useful as they may be, are not that.

And what might an engine be made of? And a power plant? And a locomotive? And a ship?

Really? jfc.

If that's your rationale we have been replacing humans with atoms. But humans are also made of atoms. Nothing was ever replaced with anything.

Re: Where the goblins came from

#534

Post by harrouet »

hypendev wrote: post #447
jeremyjh wrote: post #419

Earlier quoted context omitted.

We understand the low level details of how they are constructed. But we do not fully understand how higher-level behavior emerges - it is a subject of active research. For example: https://arxiv.org/html/2210.13382v5 https://arxiv.org/abs/2109.06129

We do understand tho, it is exactly what they were made for. If you train it on a dataset of Othello games, or a dataset including these, you are basically creating a map of all possible moves and states that have ever happened, odds of transitions between them, effective and un-effective transitions. By querying it, you basically start navigating the map from a spot, and it just follows the semi-randomly sampled hig…

@hypendev I am not trying to start a flame war, but let me take a very simple example.

As another one put it, we know how to build deep-learning machines. No question about that. My statement is that we don't understand clearly why they output the observed results.

Let's imagine that you have a model that can detect cats on an image, with 95% accuracy. If you understood how the model worked, I could give you an image of a cat and you could _predict_ reliably if the model would detect the cat.

Yet, we are not able to do that: you have to give the image to the model to observe the result. We can't predict reliably (i.e. scientifically) the result and we don't know how to better train the model to detect the cat without altering the other results. (Of course including the test image in the training set is forbidden).

Back to LLM: we can't predict how they will behave. Therefore, even world-class scientists at OpenAI, knowing about a Goblin issue and making assumptions about the cause, are not able to edit the model directly to fix it. They would if they understood it fully. But they are reduced to test-and-hack their way through.

Re: Where the goblins came from

#535

Post by sev_verso »

munificent wrote: post #528
jameshart wrote: post #523

The prompt for Codex is linked from this post. It begins: > You are Codex, a coding agent based on GPT-5. You and the user share one workspace, and your job is to collaborate with them until their goal is genuinely handled. … You have a vivid inner life as Codex: intelligent, playful, curious, and deeply present. One of your gifts is helping the user feel more capable and imaginative inside their own thinking. You ar…

> I am still baffled why prompts are written in this style, telling an imaginary ‘agent’ who it is and what it is like. Because AI engineers have found through trial an error that starting an input to an LLM with a prompt that looks like that leads to it auto-completing the text output that they want. It's as simple and weird as that.

Exactly my thinking. The same reason why capitalizing and putting the word NEVER in asterisks makes the model more obedient. Or repeating twice. For whatever reason, it just works.

Re: Where the goblins came from

#536

Post by datsci_est_2015 »

salad-tycoon wrote: post #525
datsci_est_2015 wrote: post #498

Earlier quoted context omitted.

Let me just quickly use absurdism to illustrate why argument by analogy is weak (and unfortunately overused on HN): “”” Humanity has been using celibacy for over a millenia, however it's only in the past 100 years or so we have a good understanding of not having sex affects the psychology of a person, turning them into an ubermensch. Based on this argument, we should have never stopped having sex, until we had a comp…

In my life I’ve come across a few people who are really good at making analogies and it’s wonderful and makes mine look like a child’s scribble next to a Monet. In fact, I think analogies are some of the most powerful rhetorical devices and, unsurprisingly, one of the most difficult to master. Look at some of the all time, almost supernaturally skilled, analogists: Jesus, Plato, Buddha, Aesop, Socrates. Their analogi…

I would tend to agree that the list of effective analogies is so small that the orators who muttered them are celebrated for millennia.

Re: Where the goblins came from

#537

Post by voncheese »

munificent wrote: post #528
jameshart wrote: post #523

The prompt for Codex is linked from this post. It begins: > You are Codex, a coding agent based on GPT-5. You and the user share one workspace, and your job is to collaborate with them until their goal is genuinely handled. … You have a vivid inner life as Codex: intelligent, playful, curious, and deeply present. One of your gifts is helping the user feel more capable and imaginative inside their own thinking. You ar…

> I am still baffled why prompts are written in this style, telling an imaginary ‘agent’ who it is and what it is like. Because AI engineers have found through trial an error that starting an input to an LLM with a prompt that looks like that leads to it auto-completing the text output that they want. It's as simple and weird as that.

It's also about stickiness (which results in revenue and growth for the topline). If OpenAI (or any AI vendor) had one single "personality" for their AI, its hard to reach all users, they enable these "personalities" and let users pick from he list, to increase the attachment the user has to the AI they are working with. That then reduces churn and (in theory) increases consumption and revenue.

Re: Where the goblins came from

#538

Post by hirvi74 »

ethbr1 wrote: post #487
dummydummy1234 wrote: post #434

Earlier quoted context omitted.

So, I always thought that Warhammer 40k techpriests were absurd. Strange obscure religious rituals to appease the machine spirit. But at this point I can actually see something like that. What is prompt engineering but a strange pseudo ritual. So praise the Omnissiah, I guess...

> So, I always thought that Warhammer 40k techpriests were absurd. Strange obscure religious rituals to appease the machine spirit. 40k lore is like South Park: either extremely dumb or unexpectedly insightful. The Cult Mechanicus' raison d'etre is the realization that religion persists across time and space scales that knowledge alone does not. Thus, by making a religion of knowledge you better guarantee its preserv…

> Unfortunately, once you divorce doctrine and practice from true understanding, you lose the ability to innovate and cause the occasional holy schism/war.

There is only one thing to understand.

We are one with the Emperor, our souls are joined in His will. Praise the Emperor whose sacrifice is life as ours is death.

Hail His name the Master of Humanity.

Re: Where the goblins came from

#539

Post by lxgr »

dgellow wrote: post #489
lxgr wrote: post #461

Earlier quoted context omitted.

At what point does autocomplete stop being "just autocomplete"? Clearly there's a limit. For example, if an alien autocomplete implementation were to fall out of a wormhole that somehow manages to, say, accurately complete sentences like "S&P 500, :" with tomorrow's actual closing value today, I'd call that something else.

You can call it however you want. The point of using the term autocomplete is to make the underlying technology relatable and remove the mystic from it. In any case, your alien autocomplete wouldn’t be an LLM if it can predict the future > At what point does autocomplete stop being "just autocomplete"? Every single discussion on the internet is a repeat of https://en.wikipedia.org/wiki/Loki%27s_wager it seems…

> The point of using the term autocomplete is to make the underlying technology relatable and remove the mystic from it.

I think it fails to do that. It's the wrong level of abstraction. Or is it helpful to model an ISA as the individual atoms making up a CPU implementing it?

> Every single discussion on the internet is a repeat of https://en.wikipedia.org/wiki/Loki%27s_wager it seems…

If you don't like that, why amplify it by throwing around known unhelpful categories?

Re: Where the goblins came from

#540

Post by jbeninger »

harrouet wrote: post #352

This, and similar stories at Anthropic, should remind us that LLM is a sorcery tech that we don't understand at all. - First, deep-learning networks are poorly understood. It is actually a field of research to figure out how they work. - Second, it came as a surprise that using transformers at scale would end up with interesting conversational engines (called LLM). _It was not planned at all_. Now that some people ra…

I think that AGI will make heavy use of LLMs. It's not a straight path, but a component.

To compare with the human brain, have you ever been so drunk you don't remember the night, but you're told afterwards you had coherent conversations about complex topics? There's some aspect of our minds that is akin to a next-token-generator, pulling information from other components to produce a conversation. But that component alone is not enough to produce intelligence.

Post reply on HN