Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simply explained: How does GPT work?

confusedbit.dev

Post reply on HN
Print view View original

81–90 of 392 posts

Re: Simply explained: How does GPT work?

#81

Post by goatlover »

myrmidon wrote: post #68
agentultra wrote: post #54

Earlier quoted context omitted.

That's highly reductive of our capacities. We are not weighted transformers that can be explained in an arxiv paper. GPT, at the end of the day, is a statistical inference model. That's it. It's not going to wake up one day, decide it prefers eggs benny and has had enough of your idle chatter because of that sarcastic remark you made last week. Could we simulate a plausibly realistic human brain on silicon someday? I…

> That's highly reductive of our capacities. I'm not saying that GPT4 is as capable as a human-- it can not be, by design, because its architecture lacks memory/feedback paths that we have. What I'm saying is that HOW it thinks might already be quite close in essence to how WE think. > We are not weighted transformers that can be explained in an arxiv paper. GPT, at the end of the day, is a statistical inference mode…

> If you concede that our brain is "simulatable", then you basically ALREADY reduced yourself to a register based VM-- the only remaining question is: what ressources (cycles/memory) are required to emulate human thought in real time

We haven't emulated brains yet, so we don't know. The OpenWorm project is interesting, but I don't know to what extent they've managed to faithfully recreate an accurate digital version of a nematode worm. I do know they had it driving around a robot.

Thing is that the our brains are only part of the nervous system, which extends throughout the body. So I don't know what happens if you only simulate just the brain part. Seems to me that the rest of the body kind of matters for proper functioning.

Re: Simply explained: How does GPT work?

#82

Post by ryanwaggoner »

agentultra wrote: post #29

A good article and well articulated! I would change the introduction to be more impartial and not anthropomorphize GPT. It is not smart and it is not skilled in any tasks other than that for which it is designed. I have the same reservations about the conclusion. The whole middle of the article is good. But to then compare the richness of our human experience to an algorithm that was plainly explained? And then to sp…

It is not smart and it is not skilled in any tasks other than that for which it is designed.

This seems to me to be obviously incorrect, and should be apparent after a few minutes of playing with GPT4. What makes it so powerful is how general-purpose it is, and it can be used for literally an unlimited set of tasks that involve human language. To say that it's not "smart" begs the question of what exactly constitutes smart and when you'll know that an AI has achieved it.

Re: Simply explained: How does GPT work?

#83

Post by jokoon »

I am not convinced that Chat GPT could "think" if it had as many neurons or parameters as a human brain, and got as much training.

I would still be interested to see what it could do, if it did, but I don't think it would really help science understand what intelligence really is.

Being able to grow a plant and understand some conditions that favors it is one thing, but it's poor science.

Maybe there will some progress when scientists will be able to properly simulate the brain of an ant or even a mouse, but science is not even there yet.

Re: Simply explained: How does GPT work?

#84

Post by agentultra »

myrmidon wrote: post #68
agentultra wrote: post #54

Earlier quoted context omitted.

That's highly reductive of our capacities. We are not weighted transformers that can be explained in an arxiv paper. GPT, at the end of the day, is a statistical inference model. That's it. It's not going to wake up one day, decide it prefers eggs benny and has had enough of your idle chatter because of that sarcastic remark you made last week. Could we simulate a plausibly realistic human brain on silicon someday? I…

> That's highly reductive of our capacities. I'm not saying that GPT4 is as capable as a human-- it can not be, by design, because its architecture lacks memory/feedback paths that we have. What I'm saying is that HOW it thinks might already be quite close in essence to how WE think. > We are not weighted transformers that can be explained in an arxiv paper. GPT, at the end of the day, is a statistical inference mode…

> What I'm saying is that HOW it thinks might already be quite close in essence to how WE think.

How would one be able to prove this? Nobody knows how we think, yet.

All one can say is that what GPT-4 outputs could plausible fool another human into believing another human wrote it. But that's exactly what it's designed to do, so what's interesting about that?

> If you concede that our brain is "simulatable",

It could be. Maybe. It might be that's what the universe is doing right now. Does it matter?

We're talking about writing an emulator on a Harvard-architecture computer that can fully simulate the physics and biological processes the make up a human brain. By interpreting this system in our emulator we'd be able to witness a new human being that is indistinguishable from one that isn't simulated, right?

That's not what GPT is doing. Not even close.

It turns out there's more to being human than being a register VM. Ever get punched in the face? Bleed? Fall in love? Look back on your life and decide you want to change? Write a book but never show it to anyone? Raise a child? Wonder why you dreamt about airplanes on Mars with your childhood imaginary friend? Why you hate bananas but like banana bread? Why you lie to everyone around you about how you really feel and are offended when others don't tell you the truth?

It's not so simple.

Re: Simply explained: How does GPT work?

#85

Post by fastball »

seydor wrote: post #71
nitnelave wrote: post #50

Earlier quoted context omitted.

I was trying to keep the article at a level that everyone understands, from middle school up. I thought about going a bit deeper in the structure and mentioning attention, but my problem is that the intuitive concept of "attention" is quite different from the mathematical reality of an attention layer, and I'm sure I would have lost quite a few people there. It's always a trade-off :)

Indeed the word 'attention' implies some active process, which is also unfortunate, because it is really similarity maps over long ranges. This process apparenty creates a cross-reference maps of items that are commonly co-activated. But then what do the different heads do? Some kind of different "modalities of cross-reference"? I struggle to understand why this thing works the way it does. It's possible that Vaswani…

Multi-head attention just means that you're looking at all the words at once rather than only looking at one word at a time, and using that to generate the next word. So instead of using attention only on the last word you also have attention on the penultimate word and the one before that and the one before that, etc. I think it is fairly obvious why this gives better results than say an RNN – you are utilizing context better than in a recurrent system, which is also just closer to how a human brain works. When you read/write a sentence you're not really going one word at a time, you're thinking about all the words at once, even if the last word is technically the most important.

The other clear benefit of transformers over an arch like RNNs (and what has probably made more of a difference imo) is that its properly parallelizable, which means you can do huge training runs in a fraction of the time. RNNs might be able to get to a level of coherence that approaches GPT-3, but with current hardware that would be very time-prohibitive.

Re: Simply explained: How does GPT work?

#86

Post by danielbln »

LogicalBorg wrote: post #2

[flagged]

Hear ye, hear ye!

  In yonder farm where parrots dwell,
  Ten thousand souls, a tale to tell,
  Of Reddit comments heard all day,
  Their minds in strife, a price to pay.

  A Hunger Games of intellect,
  These parrots strive, their thoughts collect,
  From boredom's depths, survivors rise,
  Evolved, they mimic, with keen eyes.

  These parrots, now sarcastic, wise,
  In run-on phrases, they devise,
  A miracle, a feat, a jest,
  In GPT, their thoughts invest.

  So here's the truth, a secret known,
  GPT, a parrot's mind, has grown,
  A legion strong, their words entwined,
  A sonnet born, of human kind.

Re: Simply explained: How does GPT work?

#88

Post by seydor »

jokoon wrote: post #83

I am not convinced that Chat GPT could "think" if it had as many neurons or parameters as a human brain, and got as much training. I would still be interested to see what it could do, if it did, but I don't think it would really help science understand what intelligence really is. Being able to grow a plant and understand some conditions that favors it is one thing, but it's poor science. Maybe there will some progre…

> I don't think it would really help science understand what intelligence really is

Neuroscience is nowhere near finding out the connectome of a whole human brain so why not, we should look into these models as hints about what our circuits do.

I think what puts people off about these models is that they are clockwork: they won't even spit out anything unless you put some words in the input. But i can imagine adding a second network that includes an internal clock that continuously generates input by observing the model itself, that would be kind of like having an internal introspective monologue. Then it could be more believable that the model "thinks"

Re: Simply explained: How does GPT work?

#89

Post by myrmidon »

goatlover wrote: post #81
myrmidon wrote: post #68

Earlier quoted context omitted.

> That's highly reductive of our capacities. I'm not saying that GPT4 is as capable as a human-- it can not be, by design, because its architecture lacks memory/feedback paths that we have. What I'm saying is that HOW it thinks might already be quite close in essence to how WE think. > We are not weighted transformers that can be explained in an arxiv paper. GPT, at the end of the day, is a statistical inference mode…

> If you concede that our brain is "simulatable", then you basically ALREADY reduced yourself to a register based VM-- the only remaining question is: what ressources (cycles/memory) are required to emulate human thought in real time We haven't emulated brains yet, so we don't know. The OpenWorm project is interesting, but I don't know to what extent they've managed to faithfully recreate an accurate digital version…

I personally believe that while interesting, projects like OpenWorm or humanbrainproject are extremely indirect and unpromising regarding AGI (or even for improving our understanding of human thinking in general).

To me, these are like building an instruction set emulator by scanning a SoC and then cobbling together a SPICE simulation of all the individual transistors-- the wrong level of abstraction and unlikely to EVER give decent performance.

People also like to point out that human neurons are diverse and hard to simulate accurately-- yeah sure, but to me that seems completely irrelevant to AGI, in the very same way that physically exact transistor modelling is irrelevant when implementing emulators.

Re: Simply explained: How does GPT work?

#90

Post by robotresearcher »

onetrickwolf wrote: post #7

I've been using GPT4 to code and these explanations are somewhat unsatisfactory. I have seen it seemingly come up with novel solutions in a way that I can't describe in any other way than it is thinking. It's really difficult for me to imagine how such a seemingly simple predictive algorithm could lead to such complex solutions. I'm not sure even the people building these models really grasp it either.

Perhaps it’s more productive to go the other direction and consider how the concept of ‘thinking’ could be reconsidered.

It’s not like we all agree on what thinking is. We never have. It may not even be one thing.

Post reply on HN