Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simply explained: How does GPT work?

confusedbit.dev

Post reply on HN
Print view View original

141–150 of 392 posts

Re: Simply explained: How does GPT work?

#141

Post by simonw »

lm28469 wrote: post #103
simonw wrote: post #15

Earlier quoted context omitted.

I used GPT-4 to build this tool https://image-to-jpeg.vercel.app using a few prompts the other day - my ChatGPT transcript for that is here: https://gist.github.com/simonw/66918b6cde1f87bf4fc883c677351...

See my problem with virtually every single example is that we talk about "I can't describe in any other way than it is thinking", "such complex solutions" but in the end we get a 50 lines "app" that you'd see in a computer science 101 class It's very nice, it's very impressive, it will help people, but it doesn't align with the "you're just about to lose your job" "Skynet comes in the next 6 months" &c. If these basi…

This took me 3 minutes to build. Without ChatGPT it would have taken me 30-60 minutes, if not longer thanks to the research I would have needed to do into the various browser APIs.

If it had taken me longer than 3 minutes I wouldn't have bothered - it's not a tool I needed enough to put the work in.

That's the thing I find so interesting about this stuff: it's causing me to be much more ambitious in what I chose to build: https://simonwillison.net/2023/Mar/27/ai-enhanced-developmen...

Re: Simply explained: How does GPT work?

#142

Post by agentultra »

myrmidon wrote: post #117
agentultra wrote: post #102

Earlier quoted context omitted.

> I'm arguing that GPT4 is essentially the second approach Ok, so then it is an algorithm that simulates a specific behaviour that produces plausibly human-level results. My point is that this is not thinking , smart , or "general intelligence." Let's say I write an algorithm that can also produce text. It's not an implementation of the specification for GPT-4 but something novel. It takes the exact same inputs and p…

> My point is that this is not thinking, smart, or "general intelligence." Why not? I would already, without hesitation, describe GPT4 as strictly more intelligent than my cat and also all gradeschoolers I've ever known... Maybe some adults, too- depends on your exact definition of intelligence. > Let's say I write an algorithm [...], you can't tell if [input] was produced by GTP-4 or my algorithm. Sure, I'd call you…

> I would already, without hesitation, describe GPT4 as strictly more intelligent than my cat

Well if we're going to define intelligence based one what you believe it is then why don't you explain it?

I'm not the one claiming to know what intelligence is or that we can even simulate a system capable of emulating this characteristic. So if you hold the specification for human thought I think you ought to share it with us.

> Why not?

By definition. ChatGPT is designed for a single function, the description of which are the specifications and the code that implements it. Nothing in this specification implies it is capable of anything except what is described.

Calling it, "intelligent," is a mischaracterization at best and anthropomorphism at worst. The same follows for calling it "smart" or claiming it is, "skilled at X."

Re: Simply explained: How does GPT work?

#143

Post by Zetice »

danenania wrote: post #139
Zetice wrote: post #127

Does anyone have a good recommendation for a book that would cover the underlying ideas behind LLMs? Google ends up giving me a lot of ads, and ChatGPT is vague about specifics as per usual.

Not a book, but here's a really good explanation in blog post form from Stephen Wolfram: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

I do not trust that man one iota.

Re: Simply explained: How does GPT work?

#144

Post by wolpoli »

winternett wrote: post #21

Where is IBM's Watson in all this? It seems as if it never existed? That is just one example of how companies keep making these grand presentations and under-delivering on results... Plain and simple the over-hyped GPT editions are NOT truly AI, it is scripting to assemble coherent looking sentences backed by scripts that parse content off of of stored data and the open web into presented responses.... There is no "a…

> Where is IBM's Watson in all this? It seems as if it never existed?

Nowadays, IBM's Watson is simply a brand name for any AI/ML related products under IBM.

Re: Simply explained: How does GPT work?

#145

Post by tines »

iliane5 wrote: post #136
i-use-nixos-btw wrote: post #130

I’d be interested in hearing from anyone who takes the Chinese Room scenario seriously, or at least can see how it applies to any of this. I cannot see that it matters if a computer understands something. If it quacks like a duck and walks like a duck, and your only need is for it to quack and walk like a duck, then it doesn’t matter if it’s actually a duck or not for all intents and purposes. It only matters if you…

What I find really entertaining is the "just predicting the next token" argument. If just predicting the next token can produce similar or better results than the almighty human intelligence on some tasks, then maybe there's a bit of hubris in how smart we think we actually are.

> If just predicting the next token can produce similar or better results than the almighty human intelligence on some tasks

But it's not better than almighty human intelligence, it _is_ human intelligence, because it was trained on a mass of some of the best human intelligence in all recorded history (I say this because the good stuff like Aristotle got preserved while the garbage disappeared (this was true until the recent internet age, in which garbage survives as well as the gold)).

> then maybe there's a bit of hubris in how smart we think we actually are

I feel like you could say this if ChatGPT or whatever obtained its knowledge some other way than direct guidance from humans, but since we hand-fed it the answers, it falls a little flat for me.

I'm open to persuasion.

Re: Simply explained: How does GPT work?

#146

Post by seizethecheese »

Zetice wrote: post #143
danenania wrote: post #139

Earlier quoted context omitted.

Not a book, but here's a really good explanation in blog post form from Stephen Wolfram: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

I do not trust that man one iota.

The blog post is very good.

Re: Simply explained: How does GPT work?

#148

Post by tines »

danenania wrote: post #123
ZeroGravitas wrote: post #104

> It is able to link ideas logically, defend them, adapt to the context, roleplay, and (especially the latest GPT-4) avoid contradicting itself. Isn't this just responding to the context provided? Like if I say "Write a Limerick about cats eating rats" isn't it just generating words that will come after that context, and correctly guessing that they'll rhyme in a certain way? It's really cool that it can generate coh…

It's all about emergent complexity. While you can reduce it to "just" statistical auto-completion of the next word, we are seeing evidence of abstraction and reasoning produced as a higher-order effect of these simple completions. It's a bit like the Sagan quote: "If you wish to make an apple pie from scratch, you must first invent the universe". Sometimes for GPT to "just" complete the next word in a way that humans…

> Sometimes for GPT to "just" complete the next word in a way that humans find plausible, it must, along the way, develop a model of the world, theory of mind, abstract reasoning. etc.

I did an experiment recently where I asked ChatGPT to "tell me an idea [you] have never heard before". ChatGPT replied with what sounded like an idea for a startup, which was delivering farm-fresh vegetables to customers' doors. This is of course not an idea it has never heard before, it's on the internet.

If you asked a human this, they would give you an idea they had never heard before, whereas ChatGPT simply "finds" training data where someone asked a similar question, and produces the likely response, which is an idea that it has actually "heard," or seen in its training data, before. (Obviously a gross simplification of the algorithm but the point stands.)

This is a difference between ChatGPT's algorithm and human reasoning. The things that you mention, the model of the world, theory of mind, etc. are statistical illusions which have observable differences from the real thing.

Am I wrong? I'm open to persuasion.

Re: Simply explained: How does GPT work?

#149

Post by sirwhinesalot »

swframe2 wrote: post #133
sirwhinesalot wrote: post #46

It predicts the next word/token based on the previous pile of words/tokens. Given a large enough model (as in GPT3+) it can actually output some rather useful text because the probabilities it learned on what the next token should be are rather accurate.

(my opinion) It is not predicting based on 'words/tokens'. It is transforming the general words/tokens embeddings into a context specific embedding which encodes "meaning". It is not an n-gram model of words. It is more like an n-gram model of "meaning". It doesn't encode all the "meanings" that humans are able to but with addition labelled data it should get closer. I think gpt is a component which can be combined t…

You are absolutely right, that's the more in depth explanation as to why it's not just an overly complicated markov chain.

At the same time, "meaning" here is essentially "close together in a big hyperdimensional space". It's meaning in the same way youtube recommendations are conceptually related by probability.

And yet, the output is nothing short of incredible for something so blunt in how it functions, much like our brains I suppose.

I'm a die-hard classical AI fan though, I like knowing the rules and that the results are provably optimal and that if I ask for a different result I can actually get a truly meaningfully different output. Not nearly as convenient as a chat bot of course, and unfortunately ChatGPT is abysmal at generating constraint problems. Maybe one day we'll get a best of both worlds.

Re: Simply explained: How does GPT work?

#150

Post by olooney »

Zetice wrote: post #127

Does anyone have a good recommendation for a book that would cover the underlying ideas behind LLMs? Google ends up giving me a lot of ads, and ChatGPT is vague about specifics as per usual.

The 2nd edition of Deep Learning with Python[1] (by François Chollet of Keras fame) has a pretty good chapter on the transformer architecture[2]. In fact, the explanations in the book are clear and lucid across the board.

[1]: https://www.amazon.com/Deep-Learning-Python-Francois-Chollet...

[2]: https://en.wikipedia.org/wiki/Transformer_(machine_learning_...

Post reply on HN