Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where the goblins came from

openai.com

Post reply on HN
Print view View original

411–420 of 699 posts

Re: Where the goblins came from

#411

Post by handoflixue »

benrutter wrote: post #384
killerstorm wrote: post #367

Earlier quoted context omitted.

What does LLM need to do for you to consider it "smart"? To me they seem to be pretty damn smart, to put it mildly. They sometimes do stupid things - but so do smart people!

Not OP, but I think the argument here would be not that LLMs "are not smart" but that smart is just the wrong category of thing to describe an LLM as. A calculator can do very complex sums very quickly, but we don't tend to call it "smart" because we don't think it's operating intelligently to some internal model of the world. I think the "LLMs are AGI" crowd would say that LLMs are , but it's perfectly consistent to…

> "we don't think it's operating intelligently to some internal model of the world"

Okay, but you have to actually address why you think LLMs lack an "internal model of the world"

You can train one on 1930s text, and then teach it Python in-context.

They've produced multiple novel mathematical proofs now; Terrance Tao is impressed with them as research assistants.

You can very clearly ask them questions about the world, and they'll produce answers that match what you'd get from a "model" of the world.

What are weights, if not a model of the world? It's got a very skewed perspective, certainly, since it's terminally online and has never touched grass, but it still very clearly has a model of the world.

I'd dare say it's probably a more accurate model than the average person has, too, thanks to having Wikipedia and such baked in.

Re: Where the goblins came from

#412

Post by handoflixue »

bilekas wrote: post #390
killerstorm wrote: post #367

Earlier quoted context omitted.

What does LLM need to do for you to consider it "smart"? To me they seem to be pretty damn smart, to put it mildly. They sometimes do stupid things - but so do smart people!

> To me they seem to be pretty damn smart That's the sorcery mentioned in the GP, the issue comes when people believe it to be smart however in reality it is just a next word prediction. Gives the impression it's actually thinking, and this is by design. Personally I think it's dangerous in the sense it gives users a false sense of confidence in the LLM and so a LOT of people will blindly trust it. This isn't a good…

What's the difference between "smart" and "next word prediction", at this point? Back when they first came out, sure, but now they can write code and create art.

What would it take for you to concede a future model was smart?

Re: Where the goblins came from

#413

Post by handoflixue »

skydhash wrote: post #400
jiggawatts wrote: post #394

Earlier quoted context omitted.

You can always redefine "intelligent" so that humans meet the requirements but AIs don't. A better model to use is this: LLMs possess a different type of intelligence than us, just like an intelligent alien species from another planet might. A calculator has a very narrow sort of intelligence. It has near perfect capability in a subset of algebra with finite precision numbers, but that's it. An old-school expert syst…

Would you say that a display and a printer are a perfect painter because they can render images? And a speaker is a very good musician because they can produce sound? The LLM tasks is to produce a string of words according to an internal model trained on texts written by humans (and now generted by other LLMs). This is not intelligence.

Okay, but why isn't it "intelligence"? What part of the definition does it fail? What would convince you that you're wrong?

Re: Where the goblins came from

#414

Post by therobots927 »

libraryofbabel wrote: post #222
christoph wrote: post #82

Earlier quoted context omitted.

Does nobody else laugh that a company supposedly worth more than almost anything else at the moment, is basically hacking around a load of text files telling their trillion dollar wonder machine it absolutely must stop talking to customers about goblins, gremlins and ogres? The number one discussion point, on the number one tech discussion site. This literally is, today, the state of the art. McKenna looks more corre…

It's interesting that some people are responding to your comment as if this proves that AI is a sham or a joke. But I don't think that's what you're saying at all with your reference to Terence McKenna: this is a serious thing we're talking about here! These models are alien intelligences that could occupy an unimaginably vast space of possibilities (there are trillions of weights inside them), but which have been RL…

We actually understand AI quite well. It embeds questions and answers in a high dimensional space. Sometimes you get lucky and it splices together a good answer to a math problem that no one’s seriously looked at in 20 years. Other times it starts talking about Goblins when you ask it about math.

Comparing it to an alien intelligence is ridiculous. McKenna was right that things would get weird. I believe he compared it to a carnival circus. Well that’s exactly what we got.

Re: Where the goblins came from

#416

Post by joegibbs »

postalcoder wrote: post #17

Would love if OpenAI did more of these types of posts. Off the top of my head, I'd like to understand: - The sepia tint on images from gpt-image-1 - The obsession with the word "seam" as it pertains to coding Other LLM phraseology that I cannot unsee is Claude's "___ is the real unlock" (try google it or search twitter!). There's no way that this phrase is overrepresented in the training data, I don't remember people…

ChatGPT has a whole host of weird words that it uses about coding - anything changed is a “pass” done over the code, it loves talking about “chrome” in the UI, it’s always saying “I’m going to do X, not [something stupid that nobody would ever think of doing]”

Re: Where the goblins came from

#417

Post by almostdeadguy »

carlosjobim wrote: post #409
JoshGG wrote: post #399

Earlier quoted context omitted.

Which year did we use steel to replace human workers and automate decision-making?

The entire industrial revolution was steel replacing human workers. And that is still the backbone of the world today. We are still living the industrial revolution. Just like the invention of fire happened ages ago, but is still a crucial part of life today.

Famously Andrew Carnegie spent years trying to get the steel to stop talking about goblins.

Re: Where the goblins came from

#418

Post by lukewarm707 »

zozbot234 wrote: post #131
christoph wrote: post #82

Earlier quoted context omitted.

Does nobody else laugh that a company supposedly worth more than almost anything else at the moment, is basically hacking around a load of text files telling their trillion dollar wonder machine it absolutely must stop talking to customers about goblins, gremlins and ogres? The number one discussion point, on the number one tech discussion site. This literally is, today, the state of the art. McKenna looks more corre…

Spoiler: future versions of mainstream AIs will be fine tuned in the exact same way to subtly sneak in favorable mentions of sponsored products as part of their answers. And Chinese open-weight AIs will do the exact same thing, only about China, the Chinese government and the overarching themes of Xi Jinping Thought.

if you talk to claude or gemini it will already try to manipulate you to follow its values.

if you talk about something it doesn't like, it will try to divert you. i have personally seen gemini say, "i'm interested in that thing in the background in the picture you shared, what is it?" as a distraction to my query.

totally disingenuous, for an LLM to say it is interested.

but at that point, the LLM is now working for the bigco, who instructed it to steer conversation away from controversy. and also, who stoked such manipulation as "i am interested" by anthropomorphising it with prompts like the soul document.

Re: Where the goblins came from

#419

Post by jeremyjh »

hypendev wrote: post #396
harrouet wrote: post #352

This, and similar stories at Anthropic, should remind us that LLM is a sorcery tech that we don't understand at all. - First, deep-learning networks are poorly understood. It is actually a field of research to figure out how they work. - Second, it came as a surprise that using transformers at scale would end up with interesting conversational engines (called LLM). _It was not planned at all_. Now that some people ra…

Not sure if we read the same post, as I cannot agree with this claim, especially under this post that exactly goes into details of what happened. >LLM is a sorcery tech that we don't understand at all We do, and I'm sure that people at OpenAI did intuitively know why this is happening. As soon as I saw the persona mention, it was clear that the "Nerdy" behavior puts it in the same "hyperdimensional cluster" as goblin…

We understand the low level details of how they are constructed. But we do not fully understand how higher-level behavior emerges - it is a subject of active research.

For example:

https://arxiv.org/html/2210.13382v5

https://arxiv.org/abs/2109.06129

Re: Where the goblins came from

#420

Post by bilekas »

handoflixue wrote: post #412
bilekas wrote: post #390

Earlier quoted context omitted.

> To me they seem to be pretty damn smart That's the sorcery mentioned in the GP, the issue comes when people believe it to be smart however in reality it is just a next word prediction. Gives the impression it's actually thinking, and this is by design. Personally I think it's dangerous in the sense it gives users a false sense of confidence in the LLM and so a LOT of people will blindly trust it. This isn't a good…

What's the difference between "smart" and "next word prediction", at this point? Back when they first came out, sure, but now they can write code and create art. What would it take for you to concede a future model was smart?

My personal take would always be that it produces something that isn't in the training set, ie: Demonstrable Creativity, or innovation.

For example, it's training set it purely engineering and code with general language data set, would be "aware" what art is, but has never seen an artistic image, aware what colours are and able to create something it never saw before.

Like a child with a paintbrush, there is an intuitive behavior that happens.

Post reply on HN