Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simply explained: How does GPT work?

confusedbit.dev

Post reply on HN
Print view View original

151–160 of 392 posts

Re: Simply explained: How does GPT work?

#151

Post by ww520 »

Zetice wrote: post #127

Does anyone have a good recommendation for a book that would cover the underlying ideas behind LLMs? Google ends up giving me a lot of ads, and ChatGPT is vague about specifics as per usual.

ML these days have moved forward fast that books are outdated after couple years. Look at articles, podcasts, youtube videos, and research papers for the latest changes. ChatGPT utilizes existing algorithms. Its insight is scaling the weight parameters up to billions, and human feedback & guidance help a lot in finetuning the model to be user friendly to general users.

To understand LLM from ground up, the following topics would help.

  - Machine Learning basics. e.g. weight parameters being trained.  
  - Neural Net basics.  
  - Nature Language Processing basics.
  - Word vectorization, word embedding. e.g. Word2Vec.
  - Recurrent Neural Net basics.
  - LSTM model.
  - Attention and Transformer model.
  - Generative model like GAN.
  - Generative Pre-trained Transformer.
I might miss a few topics. Actually ask ChatGPT to explain each topic. See how far it goes.

Re: Simply explained: How does GPT work?

#152

Post by danenania »

simonw wrote: post #12
onetrickwolf wrote: post #7

I've been using GPT4 to code and these explanations are somewhat unsatisfactory. I have seen it seemingly come up with novel solutions in a way that I can't describe in any other way than it is thinking. It's really difficult for me to imagine how such a seemingly simple predictive algorithm could lead to such complex solutions. I'm not sure even the people building these models really grasp it either.

I've started to suspect that generating code is actually one of the easier things for a predictive text completion model to achieve. Programming languages are a whole lot more structured and predictable than human language. In JavaScript the only token that ever comes after "if " is "(" for example.

Are you using it with static types at all? With TypeScript, I've found that it's quite good at producing the imperative logic, but can struggle with types once they reach a certain level of abstraction. It's interesting that even in the realm of "structured languages", it's a lot stronger at some kinds of inference than others.

Re: Simply explained: How does GPT work?

#153

Post by robwwilliams »

masswerk wrote: post #57
seydor wrote: post #17

This is confusing, using the semantic vectors arithmetic of embeddings is not very relevant to transformers and its completely missing the word 'attention'. I don't think transformers are that difficult to explain to people , but it is hard to explain "why" they work. But i think it's important for everyone to look under the hood and know that there are no demons underneath.

> that there are no demons underneath Given that Lacan already proposed the unconscious as structured language-like more than half a century ago and described attention in his turn on Freud's impulse in favor of his concept of derive , we may say, this is pretty much where our own demons live. (I actually do think that revisiting Lacan in this context may be productive.)

I would vote for revisiting Hofstadter and perhaps even Piaget. And if we want to bother trying to define and operationalize self-consciousness—-then Guillery’s The Brain as a Tool—is a good foray into how the CNS pulls off this simple trick.

Re: Simply explained: How does GPT work?

#154

Post by autokad »

tines wrote: post #145
iliane5 wrote: post #136

Earlier quoted context omitted.

What I find really entertaining is the "just predicting the next token" argument. If just predicting the next token can produce similar or better results than the almighty human intelligence on some tasks, then maybe there's a bit of hubris in how smart we think we actually are.

> If just predicting the next token can produce similar or better results than the almighty human intelligence on some tasks But it's not better than almighty human intelligence, it _is_ human intelligence, because it was trained on a mass of some of the best human intelligence in all recorded history (I say this because the good stuff like Aristotle got preserved while the garbage disappeared (this was true until th…

chatgpt doesnt just feed us back answers we already taught it. It learned relationships and semantics so it can apply that knowledge to do something novel. For instance, I took the basic of a dream and told it to turn it into a short story. The short story wasn't bad. I said make it more exciting, it updated the story such that one of the cars exploded. I guess chatgpt learned excitement from michael bay.

Re: Simply explained: How does GPT work?

#155

Post by Zetice »

cjblack wrote: post #147
Zetice wrote: post #143

Earlier quoted context omitted.

I do not trust that man one iota.

Why?

He's got a habit of self aggrandizing, antagonism, and deception in an effort to promote himself and his brand, I worry that his explanations are designed to maximally benefit him, rather than to maximally explain the topic.

He's a brilliant man, I just don't trust him.

Re: Simply explained: How does GPT work?

#156

Post by robwwilliams »

thanatropism wrote: post #111
masswerk wrote: post #57

Earlier quoted context omitted.

> that there are no demons underneath Given that Lacan already proposed the unconscious as structured language-like more than half a century ago and described attention in his turn on Freud's impulse in favor of his concept of derive , we may say, this is pretty much where our own demons live. (I actually do think that revisiting Lacan in this context may be productive.)

We need to form some sort of guild of engineers who think Deleuze, Latour, Lacan et caterva should be read within our disciplines.

Please no! Read systems neuroscience. Like Hassabis does. Or if of a philosophical persuasion, then Dennett or Rorty.

Re: Simply explained: How does GPT work?

#157

Post by barrysteve »

i-use-nixos-btw wrote: post #130

I’d be interested in hearing from anyone who takes the Chinese Room scenario seriously, or at least can see how it applies to any of this. I cannot see that it matters if a computer understands something. If it quacks like a duck and walks like a duck, and your only need is for it to quack and walk like a duck, then it doesn’t matter if it’s actually a duck or not for all intents and purposes. It only matters if you…

I have tried multiple times to use Chatgpt to generate Unreal c++ code. It does not do.

It spits out class names for slate objects, that inherit from other slate objects. Chatgpt doesn't understand inheritance. It just guesses what might fit inside a parameter grouping, and never suggests something with the right class type.

For my use case, it has never quacked like a duck, so to speak. It never performed, the word that might cover the concept of generating output without understanding it.

We agree on the value of computers understanding versus performing... only as much you need understanding to make it perform.

Predicting words alone does not cut the mustard, some structural depth or validating maps or some new concept is needed to sure up the wild horsepower in ChatGPT.

It must understand/have structure, or at least use a crutch to get it over the finish line..

Re: Simply explained: How does GPT work?

#158

Post by coding123 »

EForEndeavour wrote: post #32
onetrickwolf wrote: post #7

I've been using GPT4 to code and these explanations are somewhat unsatisfactory. I have seen it seemingly come up with novel solutions in a way that I can't describe in any other way than it is thinking. It's really difficult for me to imagine how such a seemingly simple predictive algorithm could lead to such complex solutions. I'm not sure even the people building these models really grasp it either.

The advanced capabilities of scaled up transformer models fed oodles of training data has burdened me with pseudo-philosophical questions about the nature of cognition that I am not well equipped to articulate, and make me wish I'd studied more neuroscience, philosophy, and comp sci earlier in life. A possibly off-topic thought dump: - What is thinking, exactly? - Does human (or superhuman) thinking require conscious…

> What even is consciousness? Why is it that when you take a bunch of molecular physical laws and scale them up into a human brain, a signal pattern emerges that feels things like emotions, continuity between moments, desires, contemplation of itself and the surrounding universe, and so on?

I doubt we'll ever be able to answer this, even after we create AGI.

Re: Simply explained: How does GPT work?

#159

Post by fl7305 »

tines wrote: post #148
danenania wrote: post #123

Earlier quoted context omitted.

It's all about emergent complexity. While you can reduce it to "just" statistical auto-completion of the next word, we are seeing evidence of abstraction and reasoning produced as a higher-order effect of these simple completions. It's a bit like the Sagan quote: "If you wish to make an apple pie from scratch, you must first invent the universe". Sometimes for GPT to "just" complete the next word in a way that humans…

> Sometimes for GPT to "just" complete the next word in a way that humans find plausible, it must, along the way, develop a model of the world, theory of mind, abstract reasoning. etc. I did an experiment recently where I asked ChatGPT to "tell me an idea [you] have never heard before". ChatGPT replied with what sounded like an idea for a startup, which was delivering farm-fresh vegetables to customers' doors. This i…

> the model of the world, theory of mind, etc. are statistical illusions which have observable differences from the real thing.

Is that a falsifiable claim?

What would an empirical test look like that would determine whether the claim ("GPT-4 does not have a model of the world") is correct or not?

Re: Simply explained: How does GPT work?

#160

Post by tines »

autokad wrote: post #154
tines wrote: post #145

Earlier quoted context omitted.

> If just predicting the next token can produce similar or better results than the almighty human intelligence on some tasks But it's not better than almighty human intelligence, it _is_ human intelligence, because it was trained on a mass of some of the best human intelligence in all recorded history (I say this because the good stuff like Aristotle got preserved while the garbage disappeared (this was true until th…

chatgpt doesnt just feed us back answers we already taught it. It learned relationships and semantics so it can apply that knowledge to do something novel. For instance, I took the basic of a dream and told it to turn it into a short story. The short story wasn't bad. I said make it more exciting, it updated the story such that one of the cars exploded. I guess chatgpt learned excitement from michael bay.

(I'm going to be brusque for the sake of the argument, I very much could be wrong and I don't even know how much I believe of the argument I'm making.)

> chatgpt doesnt just feed us back answers we already taught it

True, there is some structure to the answers we already taught it that it statistically mimics as well.

> It learned relationships and semantics so it can apply that knowledge to do something novel

Can you provide an example of this novelty? I think we underestimate the depth and variety of things that humans have written about and put on the internet, and so while anything you ask ChatGPT to do might be outside of your own experience, it's highly likely that it's already been thought before and uploaded to the internet, and that ChatGPT is just parrotting back something to you that is very similar to what it has already seen.

This effect of ChatGPT having so much more experience/training data than any single human being such that it can convince any single human that it is original is an interesting one.

This is why I think, for example, that image generation will result in (a period of) "artistic inbreeding." Because there is so much that other humans have done that is outside of any individual's experience, we will accept e.g. Midjourney's output as something moving and original, when in reality it's just a slight variation on something that someone else has done before that we haven't seen.

(Again apologies for any rudeness, I respect your opinion and experiences and am enjoying the conversation.)

Post reply on HN