Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simply explained: How does GPT work?

confusedbit.dev

Post reply on HN
Print view View original

381–390 of 392 posts

Re: Simply explained: How does GPT work?

#381

Post by fl7305 »

MacsHeadroom wrote: post #379
fl7305 wrote: post #347

Earlier quoted context omitted.

> "They are the output of non-deterministic matrix multiplication operations" Just a nit-pick: Aren't neural networks and LLMs perfectly deterministic? I think you can reproduce GPT-4 perfectly if you have access to the same source code, training data, and the seeds for the random number generators that they used? As a side note, I think it'd be theoretically possible to do this on a small 8-bit microcontroller given…

Multi-core math on GPU and CPU is non-deterministic for performance and scheduling reasons. The errors are small rounding errors that maybe don't have any serious implications right now. But the larger models get and the more operations and cores it takes to train them the more the rounding errors creep up.

> Multi-core math on GPU and CPU is non-deterministic for performance and scheduling reasons.

Ok, I see what you mean.

I can see how that could be the case. It depends on how the software is designed.

Now that I looked up it, I was surprised to see that PyTorch may generate non-reproducible results: https://pytorch.org/docs/stable/notes/randomness.html

But it looks like the sources of non-determinism in PyTorch are known, and can be avoided with a lot of work and loss of performance?

And for the general case, I don't think it's impossible to write deterministic code for multi-core processors?

> The errors are small rounding errors

But rounding errors don't imply non-deterministic answers, right? Just that the answer is different from the true answer?

Calculating the square root of 2 will have a rounding error with 32-bit floating point, but are you saying that you'll get different bit patterns in your FP32 due to rounding errors?

Re: Simply explained: How does GPT work?

#382

Post by chillacy »

tines wrote: post #283
fl7305 wrote: post #234

Earlier quoted context omitted.

Thanks. To get to what I think is the core of your argument (?) > ChatGPT simply "finds" training data where someone asked a similar question, and produces the likely response, which is an idea that it has actually "heard," or seen in its training data, before. I can definitely see a scenario where we manage to build an ultra-intelligent machine that can figure out any logical puzzle we put to it, but where it still…

That does seem really impressive. But don't you think that it's pretty likely that this, or something phrased slightly differently, appeared in the training data?

One interesting way I heard to around this is by mixing human languages in the prompt which probably never appear together in any training data, and seeing that chat gpt can do still output sensible replies. That seems to imply that something unique is happening beyond token lookup, if it’s taking different languages and mapping that to the underlying information, that looks a lot more like what people call “understanding”.

Re: Simply explained: How does GPT work?

#383

Post by JieJie »

brazzy wrote: post #331
JieJie wrote: post #273

Earlier quoted context omitted.

It was a complex situation to be sure. I only shared the story to illustrate how personally people are taking these discussions. I really felt like I was being very neutral and just sharing my enthusiasm. It was entirely unwelcome, apparently. If there's a lesson to be learned it's that people's tempers over these issues may be hotter than they appear. I can barely speak with my artist friends on the issue these days…

If are is a major part of someone's identity (strongly implied by calling them an "artist"), then it's hardly surprising that they would feel threatened and react emotionally to something that massively disrupts the creation of art and the viability of making a living from it.

I'm old enough to have had my design and publishing career evaporate underneath my feet because newsprint has mostly been replaced by the internet. I'm even old enough that I was part of replacing other people's careers with technology during the desktop publishing revolution. I remember when graphics tablets became a thing and how traditional pen and brush artists cried, "unfair!"

I always felt empathy for them, even back then. I feel empathy for my artist friends as well.

How can we ever have a better future if we keep trying to prevent change from disrupting our careers? In my experience, I've seen a lot of wailing and gnashing of teeth, but these changes have always come no matter how hard people fought. On the other hand, I see people who try to adapt to these changes do quite well.

It's the part where I watch them hide their head in the sand that bothers me, not their anger. I want them to succeed. I think they have great eyes for color and line and those things are still super-valid long after the generative AI has replaced the pen and brush.

I don't want to watch my artist friends suffer. I am old enough to know that the effort to stop generative AI is almost certainly going to fail (just as all the previous efforts to prevent change failed), and it is better to light a candle than to curse the darkness.

The only constant is change.

Re: Simply explained: How does GPT work?

#384

Post by int_19h »

pegasus wrote: post #366
int_19h wrote: post #287

Earlier quoted context omitted.

I mean, to me at least, that is the definition of discovery. The exact process used to spot the pattern is an implementation detail. And yes, I agree that we really just don't know too many things. But my impression is that we're overestimating just how complicated out behavior really is.

The rabbit hole goes very deep with these questions. For example, you left out above the other half of the equation: inventions. Our creative ability. Is that just more pattern recognition? And can discovery and invention be always cleanly teased apart? Also, what humans might have access to is something that is more simple than we imagine. Mystics and philosophers have tried to point towards it. One book that discus…

I would argue that invention is the same thing, yes - identifying patterns in the environment that can be exploited for productive purposes.

FWIW I think it's not a coincidence that LSD - which kicks pattern matching capabilities of the brain into high gear, so to speak - is commonly used as a drug to boost both "creativity" and "inventiveness".

Re: Simply explained: How does GPT work?

#385

Post by krickkrack »

agentultra wrote: post #293
krickkrack wrote: post #210

Earlier quoted context omitted.

I read this and can't help but chuckle... To say that we are nowhere being able to have AGI is quite a bold statement. It was after all only a few months ago where many people also believed we were a long way away from ChatGPT-4. The confidence with which you think we are not weighted transformers or statistical inference models is also puzzling. How could you possibly know that? How do you know that that's not preci…

Ah yes, the old: you can’t prove my deity doesn’t exist argument. Puzzling that I don’t share your faith or point of view? Why? The point is to not ascribe properties attributed to a thing we know doesn’t have them. We can teach people how ChatGPT works without getting into pseudo-philosophical babble about what consciousness is and whether humans can be accurately simulated by an LLM with enough parameters.

What faith? I never made the claim you're attributing to me. Smug idiots like you are wrong all the time.

Re: Simply explained: How does GPT work?

#386

Post by giardini »

ww520 wrote: post #151
Zetice wrote: post #127

Does anyone have a good recommendation for a book that would cover the underlying ideas behind LLMs? Google ends up giving me a lot of ads, and ChatGPT is vague about specifics as per usual.

ML these days have moved forward fast that books are outdated after couple years. Look at articles, podcasts, youtube videos, and research papers for the latest changes. ChatGPT utilizes existing algorithms. Its insight is scaling the weight parameters up to billions, and human feedback & guidance help a lot in finetuning the model to be user friendly to general users. To understand LLM from ground up, the following…

Or you could save some time by clicking on:

"What Is ChatGPT Doing … and Why Does It Work?"

https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

graciously provided above in this discussion by danenania.

As seizethecheese asserts, also above, "The blog post is very good."

Re: Simply explained: How does GPT work?

#387

Post by chlorion »

fl7305 wrote: post #380
chlorion wrote: post #377

Earlier quoted context omitted.

>So you're not different from a computer. Both are just physics. Well I do actually believe this! To me it's the only logical thing. The laws of physics apply equally to a brain and a computer, one is just a lot more fancy than the other one. >Don't you see the big difference between "I have to code the algorithms" and "the computer does it for me"? I do see the difference and understand what you are getting at. I ag…

> Well I do actually believe this! To me it's the only logical thing. The laws of physics apply equally to a brain and a computer, one is just a lot more fancy than the other one. I agree 100% with this statement taken in isolation. I don't believe there's more to a human brain than physics. There is an interesting theory about the brain using quantum mechanics ( https://en.wikipedia.org/wiki/Quantum_mind ), but that…

Wow this is a really good reply. Thanks for taking the time to write all of this!

I think I agree with pretty much all that you have said here actually, this is one of the better and more accurate descriptions of the current state of things that I have read in general!

As far as the magic thing goes, I was replying to this specifically, and other similar statements made in other parts of the thread and even in the original post (the blog post or whatever you call it) itself, and even more so in the media:

>But it wasn't designed. It's not a computer program, where one can make confident predictions about its limitations based on the source code.

There have been media headlines about the potential for modern AIs to turn evil and destroy the human race sci-fi movie style. I think people who believe this do believe that current AI is "magic" in some sense but I'm not totally sure how to pin down exactly how.

Re: Simply explained: How does GPT work?

#389

Post by samstave »

samstave wrote: post #114
ZeroGravitas wrote: post #104

> It is able to link ideas logically, defend them, adapt to the context, roleplay, and (especially the latest GPT-4) avoid contradicting itself. Isn't this just responding to the context provided? Like if I say "Write a Limerick about cats eating rats" isn't it just generating words that will come after that context, and correctly guessing that they'll rhyme in a certain way? It's really cool that it can generate coh…

There once was a Cat in New York Who got caught for feeding some Rats ; Tremendous Work! All the people tell me, many men, biggly men - many with tears in their eyes... That I have done nothing legally-wise But the truth is ; I am an enormous dork. >>_ Created by an actual Human Being with actual DNA for crime scene evidence._ - But just when they tried to brush under a rug To try to make the folks 'shrug' Is the Str…

There once was a man in New York

Guilty of paying too much for pork

He thought he would never stand

on a trial from the local grand

but corruption was just part of the work.

Re: Simply explained: How does GPT work?

#390

Post by fl7305 »

chlorion wrote: post #387
fl7305 wrote: post #380

Earlier quoted context omitted.

> Well I do actually believe this! To me it's the only logical thing. The laws of physics apply equally to a brain and a computer, one is just a lot more fancy than the other one. I agree 100% with this statement taken in isolation. I don't believe there's more to a human brain than physics. There is an interesting theory about the brain using quantum mechanics ( https://en.wikipedia.org/wiki/Quantum_mind ), but that…

Wow this is a really good reply. Thanks for taking the time to write all of this! I think I agree with pretty much all that you have said here actually, this is one of the better and more accurate descriptions of the current state of things that I have read in general! As far as the magic thing goes, I was replying to this specifically, and other similar statements made in other parts of the thread and even in the or…

Thanks, I appreciate it!

> There have been media headlines about the potential for modern AIs to turn evil and destroy the human race sci-fi movie style.

Yeah, the public debate isn't very balanced in either direction.

Post reply on HN