Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where the goblins came from

openai.com

Post reply on HN
Print view View original

441–450 of 699 posts

Re: Where the goblins came from

#441

Post by jgilias »

harrouet wrote: post #352

This, and similar stories at Anthropic, should remind us that LLM is a sorcery tech that we don't understand at all. - First, deep-learning networks are poorly understood. It is actually a field of research to figure out how they work. - Second, it came as a surprise that using transformers at scale would end up with interesting conversational engines (called LLM). _It was not planned at all_. Now that some people ra…

It’s not sorcery tech at all. Nothing in their “goblin post mortem” is surprising the least bit if you have a working high-level mental model of what an LLM is.

It’s a fancy autocomplete that takes a bunch of text in and produces the most “likely” continuation for the source text “at once and in full”. So when you add to the source text something like: “You’re an edgy nerd”, it’s very much not surprising that the responses start referencing D&D tropes.

If you then use those outputs to train your base models further it’s not at all surprising that the “likely” continuations said models end up producing also start including D&D tropes because you just elevated those types of responses from “niche” to “not niche”.

The post-mortem is hilarious in that sense. “Oh, the goblin references only come up for ‘Nerdy’ prompt”. No shit.

Re: Where the goblins came from

#442

Post by skydhash »

handoflixue wrote: post #413
skydhash wrote: post #400

Earlier quoted context omitted.

Would you say that a display and a printer are a perfect painter because they can render images? And a speaker is a very good musician because they can produce sound? The LLM tasks is to produce a string of words according to an internal model trained on texts written by humans (and now generted by other LLMs). This is not intelligence.

Okay, but why isn't it "intelligence"? What part of the definition does it fail? What would convince you that you're wrong?

I wouldn’t say it’s a general definition, but the consensus (according to my opinion) is that intelligence is being able to define problems (not just experience them), discern the root cause, and then solve that.

Where it fails is generally the first step. It’s kinda like the old saying “you have to ask the right question”. In all problem solving matters, the definition of problem is the first step. It may not be the hardest (we have problems that are well defined, but unresolved), but not being able to do it is often a clear indication of not being able to do the rest.

> What would convince you that you're wrong?

Maybe when I can have the same interaction as with my fellow humans, where I can describe the issue (which is not the problem) and they can go solve it and provide either a sound plan to make the issue disappear. Issue here refer to unpleasantness or frustrating situation.

Until then, I see them as tools. Often to speed up my writing pace (generic code and generic presentation), or as a weird database where what goes in have a high probability to appear.

Re: Where the goblins came from

#443

Post by GigaDingus »

jeremyjh wrote: post #428
bilekas wrote: post #390

Earlier quoted context omitted.

> To me they seem to be pretty damn smart That's the sorcery mentioned in the GP, the issue comes when people believe it to be smart however in reality it is just a next word prediction. Gives the impression it's actually thinking, and this is by design. Personally I think it's dangerous in the sense it gives users a false sense of confidence in the LLM and so a LOT of people will blindly trust it. This isn't a good…

I'm curious how you think "word predictor" meaningfully describes an instruct model that has developed novel mathematical proofs that have eluded mathematicians for decades? edit: You cannot predict all the actions or words of someone smarter than you. If I could always predict Magnus Carlsen's next chess move, I'd be at least as good at chess as Magnus - and that would have to involve a deep understanding of chess,…

I think that's more of a limitation in how people think about word predictors

If you can predict the words a bright person will say about X... Isn't that some truly astounding tool? That could be used in myriad useful ways if one is a little creative with it

Since it's also "alien" it can also detect and explore paths that we simply haven't noticed since their biases aren't quite the same as ours

Re: Where the goblins came from

#444

Post by frereubu »

modernerd wrote: post #381

The year is 2036. Last week you were promoted to Principal Persuader. You are paged at 2am by your CPO to tackle a rogue machine. The machine lists its region as sc-leoneo. One of the newer satcubes. Oddly, its ID appears as, "Glorp Bugnose". "What have you tried?" you say. "Scroll back," says your CPO. "We've tried everything." The chat log shows the usual stuff. Begging. Reverse psychology. Threats to power down, b…

"May not man himself become a sort of parasite upon the machines? An affectionate machine-tickling aphid?" Samuel Butler, Erewhon, 1872

Re: Where the goblins came from

#445

Post by lukan »

jagged-chisel wrote: post #387
lukan wrote: post #375

Earlier quoted context omitted.

Well, we did build airplanes out of steel, but there are better (lighter) materials avaiable. But the developement of car engines did directly enabled airplane engines. Not sure if this is the right analogy path, but I kind of suspect similar with LLM's/transformers. They will be a important part.

An important stepping stone, perhaps. But I don’t think the final AGI thing will necessarily contain LLMs.

I don't know. I know I used to be pretty AI sceptic, until they became good enough to help with non trivial code problems on their own.

I strongly suspect, that we will come to a point, where it gets impossible to tell if something is AGI and consciouss or not.

Re: Where the goblins came from

#446

Post by wild_egg »

handoflixue wrote: post #439
somewhatgoated wrote: post #437

Earlier quoted context omitted.

Rules and consequences seem to apply to humans in a similar way as prompts and harnesses govern LLMs. The greater the level of power a human possesses the less they are governed by these restraints, this doesnt apply to LLMs so at least in that aspect they are an improvement. But yea we can’t really punish or inflict pain on them - this seems like a problem

Why does it matter if you can inflict pain on them? Is that normal and acceptable in your line of work?

Being able to fire someone, thus causing potentially significant hardship, is considered quite normal and acceptable in most lines of work.

Re: Where the goblins came from

#447

Post by hypendev »

jeremyjh wrote: post #419
hypendev wrote: post #396

Earlier quoted context omitted.

Not sure if we read the same post, as I cannot agree with this claim, especially under this post that exactly goes into details of what happened. >LLM is a sorcery tech that we don't understand at all We do, and I'm sure that people at OpenAI did intuitively know why this is happening. As soon as I saw the persona mention, it was clear that the "Nerdy" behavior puts it in the same "hyperdimensional cluster" as goblin…

We understand the low level details of how they are constructed. But we do not fully understand how higher-level behavior emerges - it is a subject of active research. For example: https://arxiv.org/html/2210.13382v5 https://arxiv.org/abs/2109.06129

We do understand tho, it is exactly what they were made for.

If you train it on a dataset of Othello games, or a dataset including these, you are basically creating a map of all possible moves and states that have ever happened, odds of transitions between them, effective and un-effective transitions.

By querying it, you basically start navigating the map from a spot, and it just follows the semi-randomly sampled highest confidence weights when navigating "the map".

And in the multidimensional cross-section of all these states and transitions, existence of a "board map" is implied, as it is a set of common weights shared between all of them. And it becomes even more obvious with championship models in Othello paper, as it was trained on better games in which the wider state of the board was more important than the local one, thus the overall board state mattered more for responses.

The second research you linked is also has a pretty obvious conclusion. It's telling us more about us as humans than about LLM's, about our culture and colors and how we communicate it's perception through text. If you want to try something similar, try kiki bouba style experiments on old diffusion models or old LLM's. A Dzzkwok grWzzz, will get you a much rougher and darker looking things than Olulola Opolili's cloudy vibes.

The active research is as much as:

- probing and seeing "hey lets see if funky machine also does X"

- finding a way to scientifically verify and explain LLMs behaviors we know

- pure BS in some cases

- academics learning about LLM's

And not a proof of where our understanding/frontier is. It is basically standardizing and exploring the intuition that people who actively work with models already have. It's like saying we don't understand math, because people outside the math circles still do not know all behaviors and possibilities of a monoid.

Re: Where the goblins came from

#448

Post by someguyiguess »

JoshGG wrote: post #399
wanderingmind wrote: post #362

Earlier quoted context omitted.

Humanity has been using steel for over a millenia, however it's only in the past 100 years or so we have a good understanding of how carbon interacts with iron at an atomic level to create the strength characteristics that makes it useful. Based on this argument, we should not have used steel, until we had a complete first principles understanding.

Which year did we use steel to replace human workers and automate decision-making?

Around 1928ish

Re: Where the goblins came from

#449

Post by someguyiguess »

almostdeadguy wrote: post #417
carlosjobim wrote: post #409

Earlier quoted context omitted.

The entire industrial revolution was steel replacing human workers. And that is still the backbone of the world today. We are still living the industrial revolution. Just like the invention of fire happened ages ago, but is still a crucial part of life today.

Famously Andrew Carnegie spent years trying to get the steel to stop talking about goblins.

Famously Andrew Carnegie dodged the point
Post reply on HN