Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where the goblins came from

openai.com

Post reply on HN
Print view View original

491–500 of 699 posts

Re: Where the goblins came from

#491

Post by bel8 »

swader999 wrote: post #57
xerox13ster wrote: post #23

Earlier quoted context omitted.

Anthro means human and these are not human. Please do not use anthropology or any derivative of the word to refer to non-human constructs. I suggest Synthetipologists, those who study beings of synthetic origin or type, aka synthetipodes, just as anthropologists study Anthropodes

It is not in any sense of the word a being, it's a sophisticated generator that relies entirely on what you feed it.

> a sophisticated generator that relies entirely on what you feed it

that's me!

Re: Where the goblins came from

#492

Post by [deleted] »

hansmayer wrote: post #464
wanderingmind wrote: post #362

Earlier quoted context omitted.

Humanity has been using steel for over a millenia, however it's only in the past 100 years or so we have a good understanding of how carbon interacts with iron at an atomic level to create the strength characteristics that makes it useful. Based on this argument, we should not have used steel, until we had a complete first principles understanding.

Oh for crying out loud! Let's stop inventing fake analogies to justify the inherent LLM shortcomings! Those of us who are critical - are only using the standards that the LLM companies set themselves ("superintelligence", "pocket phds" bla blabla), to hold them accountable. When does the grift stop?

[deleted]

Re: Where the goblins came from

#493

Post by pheaded_while9 »

bko wrote: post #471
ninalanyon wrote: post #380

Earlier quoted context omitted.

> Why wouldn't you want AI to promote your countries values? Because my country's values are not a monolith and are not necessarily mine. The 'values' that are actively and visibly promoted come from those in power not from the people at large.

Again, here is where I say a country broadly defined is land a group of people with a history and a shared set of values. Politicians or rich people can't control values. They can try to impact them. But it's out of their control as its organic. The good news for you is that there is competition in AI models. So if you don't want American values and instead want Chinese or Saudi values, there will be a model to serve…

Where you are wrong is about controlling values. Axioms, incentives, and rhetorical framing are not "organic" in that they happen without a controlling force. See Prussian education, Rockefeller medicine, and your good ol' idiot box.

Re: Where the goblins came from

#494

Post by luke-stanley »

ninjagoo wrote: post #187

The level of detail they had to delve into in order to understand what was happening is wild! Apparently these systems are now complex enough to potentially justify the study of them as its own field of study [1]. The quanta article referenced at [1] used the term "Anthropologist of Artificial Intelligence"; folks appear to have issues [2] with the use of 'anthro-' since that means human. Submitted these alternative…

It's a funny detail to skim, but what's more surprising is how mechanistic interpretability and alignment science have much better tools and research than the goblin blog post suggests, including from OpenAI's own alignment team:

https://alignment.openai.com/argo/ (finding what the reward models are actually encouraging) https://alignment.openai.com/sae-latent-attribution/ (what model features drive specific behaviours, presumably this would be great for goblin hunts) https://alignment.openai.com/helpful-assistant-features/ (how high level misaligned personality shows up when fine-tuning on bad advice).

It's weird that the goblin post doesn't seem to draw upon these tools.

Anthropic's recent emotions paper shows how broad the functional emotions are, even finding specific emotions firing before cheating (!): https://transformer-circuits.pub/2026/emotions/index.html

I hope their alignment researchers aren't too annoyed by the Goblin post, it seems oddly siloed!

Re: Where the goblins came from

#496

Post by derektank »

ethbr1 wrote: post #487
dummydummy1234 wrote: post #434

Earlier quoted context omitted.

So, I always thought that Warhammer 40k techpriests were absurd. Strange obscure religious rituals to appease the machine spirit. But at this point I can actually see something like that. What is prompt engineering but a strange pseudo ritual. So praise the Omnissiah, I guess...

> So, I always thought that Warhammer 40k techpriests were absurd. Strange obscure religious rituals to appease the machine spirit. 40k lore is like South Park: either extremely dumb or unexpectedly insightful. The Cult Mechanicus' raison d'etre is the realization that religion persists across time and space scales that knowledge alone does not. Thus, by making a religion of knowledge you better guarantee its preserv…

Unfortunately, I think Vernor Vinge scooped you any way. One of the main characters of A Deepness in the Sky was something akin to a software archaeologist (I swear that exact phrase was used, but it’s been a minute) and that book was published in 1999.

Re: Where the goblins came from

#497

Post by luke-stanley »

maxdo wrote: post #2

article : bla blah blah, marketing... we are fun people, bla blah, goblin, we will not destroy the world you live in.. RL rewards bug is a culprit. blah blah.

Yeah, though it's not great marketing. Especially for hiring interpretability researchers. Their own alignment research has reward model interpretability, personality features and so on (see https://alignment.openai.com ). It just seems like a different department wrote it, which is a shame because I'd love to read about goblin feature vectors and functional emotions.

Re: Where the goblins came from

#498

Post by datsci_est_2015 »

idiotsecant wrote: post #488
ashtonshears wrote: post #475

Earlier quoted context omitted.

Poor correlation comparing physical material to computer technology

Why

Let me just quickly use absurdism to illustrate why argument by analogy is weak (and unfortunately overused on HN):

“”” Humanity has been using celibacy for over a millenia, however it's only in the past 100 years or so we have a good understanding of not having sex affects the psychology of a person, turning them into an ubermensch. Based on this argument, we should have never stopped having sex, until we had a complete first principles understanding. “””

Analogies can produce a lot of words, making it appear to be a high effort comment, but it also shifts the argument to why or why not an analogy is good or not, and away from the points the original poster was trying to make. And, by Sturgeon’s Law, most analogies are utter crap on top of being an already weak way to form an argument.

Re: Where the goblins came from

#499

Post by surgical_fire »

wanderingmind wrote: post #362
harrouet wrote: post #352

This, and similar stories at Anthropic, should remind us that LLM is a sorcery tech that we don't understand at all. - First, deep-learning networks are poorly understood. It is actually a field of research to figure out how they work. - Second, it came as a surprise that using transformers at scale would end up with interesting conversational engines (called LLM). _It was not planned at all_. Now that some people ra…

Humanity has been using steel for over a millenia, however it's only in the past 100 years or so we have a good understanding of how carbon interacts with iron at an atomic level to create the strength characteristics that makes it useful. Based on this argument, we should not have used steel, until we had a complete first principles understanding.

This is a very low-effort argument.

Humans could understand properties of steel long before they knew how Carbon interacted with Iron. Steel always behaved in a predictable, reproducible way. Empirical experiments with steel usage yielded outputs that could be documented and passed along. You could measure steel for its quality, etc.

The same cannot be said of LLMs. This is not to say they are not useful, this was never the claim of people that point at it's nondeterministic behavior and our lack of understanding of their workings to incorporate them into established processes.

Of course the hype merchants don't really care about any of this. They want to make destructive amounts of money out of it, consequences be damned.

Re: Where the goblins came from

#500

Post by gus_massa »

almostdeadguy wrote: post #417
carlosjobim wrote: post #409

Earlier quoted context omitted.

The entire industrial revolution was steel replacing human workers. And that is still the backbone of the world today. We are still living the industrial revolution. Just like the invention of fire happened ages ago, but is still a crucial part of life today.

Famously Andrew Carnegie spent years trying to get the steel to stop talking about goblins.

Steel is almost magic. Stainless steel is beyond magic.

I had a specialization in Chemistry in High School. For some analysis, the fist step is to dissolve everything in boiling Nitric Acid. But stainless steel has Chrome is like a spell of protection, so you must use boiling Hydrochloric Acid instead. I have no idea why. It's just like magic. It may have Nickel, Molybdenum, and other metals, that give it more magical properties.

A few years ago there was a nice post about copying a normal steel alloy for knives to get an equivalent made of stainless steel. You need to reduce the the Carbon content to make it less brittle. And they had to add Vanadium so it keeps the sharpness of the knives. I have no idea why. It's just like magic.

If you have half an hour, it's worth reading, but beware that it has too many technical details that are close to magical https://knifesteelnerds.com/2021/03/25/cpm-magnacut/ (HN discussion https://news.ycombinator.com/item?id=29696120 | 375 points | Dec 2021 | 108 comments)

Post reply on HN