Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT Explained: A normie's guide to how it works

jonstokes.com

Post reply on HN
Print view View original

101–110 of 144 posts

Re: ChatGPT Explained: A normie's guide to how it works

#101

Post by theturtletalks »

rvba wrote: post #98
spion wrote: post #54

Earlier quoted context omitted.

Not just connect something. For example, at a certain size, GPT models start to "learn" how to do basic arithmetic (addition), even for numbers with multiple digits they've never encountered before. It might look like a small thing, what with computers being quite able to do arithmetic at the base level. But this is a language model, so its a bit different. It learns how to add numbers without "carrying the 1" first,…

I wonder if someone could make some standardized way to describe/write mathematical proofs - in a very formal language and then train a model that will try to find more proofs for open questions.

What if it solves the P versus NP problem?

Re: ChatGPT Explained: A normie's guide to how it works

#103

Post by Kiro »

krainboltgreene wrote: post #99
spion wrote: post #54

Earlier quoted context omitted.

Not just connect something. For example, at a certain size, GPT models start to "learn" how to do basic arithmetic (addition), even for numbers with multiple digits they've never encountered before. It might look like a small thing, what with computers being quite able to do arithmetic at the base level. But this is a language model, so its a bit different. It learns how to add numbers without "carrying the 1" first,…

Okay but that's not "learning" how to do basic math in the same way I can't "learn" japanese just by mimicking the mouth noises. Yeah I'll get some of the pronunciation right sometime, maybe even get it in the right order, but only for the listener . To me, it's still just mouth noises.

Did we read the same comment? What spion describes would be complete science-fiction just a couple of years ago and yet your reaction is "meh".

Re: ChatGPT Explained: A normie's guide to how it works

#104

Post by Jensson »

Kiro wrote: post #103
krainboltgreene wrote: post #99

Earlier quoted context omitted.

Okay but that's not "learning" how to do basic math in the same way I can't "learn" japanese just by mimicking the mouth noises. Yeah I'll get some of the pronunciation right sometime, maybe even get it in the right order, but only for the listener . To me, it's still just mouth noises.

Did we read the same comment? What spion describes would be complete science-fiction just a couple of years ago and yet your reaction is "meh".

Everyone things this tech is cool, the main question discussed is how close it is to general intelligence. The poster you respond to argues that it is very far from general intelligence still, there are many who argues that it basically is general intelligence so he isn't arguing against a strawman.

If you told me "We have made a natural language parser/processor that is at human level" I'd think that was a huge step forward for computing. Nobody can argue with that.

Re: ChatGPT Explained: A normie's guide to how it works

#105

Post by Retric »

qingdao99 wrote: post #100
krainboltgreene wrote: post #99

Earlier quoted context omitted.

Okay but that's not "learning" how to do basic math in the same way I can't "learn" japanese just by mimicking the mouth noises. Yeah I'll get some of the pronunciation right sometime, maybe even get it in the right order, but only for the listener . To me, it's still just mouth noises.

If you were given the task "mimic the sound of Japanese as best you can", at first, you would just learn the basic phonology and just try and mimic the general sounds of the language. You would get good at that, and eventually you would be perfectly mimicking the pronunciation of Japanese phonemes. At some point, it would become worth your time to actually learn how different sounds are put together, e.g. Japanese ca…

Where that argument falls apart is when you’re talking about something that’s still bad a mimicry. ChatGPT still completely fails the touring test it’s not even vaguely close to that threshold.

For now it’s slightly better than fake crowd noise you’re hearing in movies, but still frequently just gibberish.

Re: ChatGPT Explained: A normie's guide to how it works

#106

Post by ChatGTP »

zerocrates wrote: post #94
ChatGTP wrote: post #73

Earlier quoted context omitted.

Bit of context might be nice.

I assume they're referring to Jorge Luis Borges's short story "The Library of Babel," which imagines an enormous library that has a book for every possible sequence of letters and punctuation, all of a certain length (a few hundred pages). The library therefore contains all useful books on any topic in any language (as long as it's covered by the alphabet), but also all useless or inaccurate ones, and of course a vas…

Thank you :)

Re: ChatGPT Explained: A normie's guide to how it works

#107

Post by iamflimflam1 »

iamflimflam1 wrote: post #62

I’m not quite sure this article is really going to help “normies”. It seems to fall into the classic trap of “I’m going to explain something in simple terms that you will understand. But, I still want you to think I’m really clever so I’m going to make sure you still don’t really understand”

Had an attempt at writing for “normies” - https://atomic14.com/2023/03/08/why-does-chatgpt-make-mistak...

Re: ChatGPT Explained: A normie's guide to how it works

#108

Post by soheil »

> Important implication: It’s common but not necessarily always useful to say the language model “knows”

What does the author think knowing actually is if not a convergence of probability distributions?

I'd like for all these "let me explain to you how things work" type write ups to first define what their understanding of the key preexisting concepts are (like what it is to know something). So when they try to explain the new concepts I don't waste my time learning concepts formed on a flawed understanding of their foundational counterparts.

Re: ChatGPT Explained: A normie's guide to how it works

#109

Post by Jensson »

IanCal wrote: post #96
Jensson wrote: post #85

Earlier quoted context omitted.

A better metaphor would be to say it compresses the internet, creates a Markov chain based on that compression. Then to make it work it compresses your prompt so that it can find it in the markov chain, move to the next step, and make a lossy decompression into a text token and adds it. The lossy decompression here is the temperature, higher temperature more lossy and more random words, but since it is lossy in the "…

> creates a Markov chain based on that compression I dislike that interpretation. It suggests it builds a very basic statistical model, but a very basic statistical model simply wouldn't be able to do what these models can do. Or alternatively, if you want to consider the model as a markov chain mapping the probability from the previous four thousand tokens to the next token then the space is astronomically large. Be…

> but a very basic statistical model simply wouldn't be able to do what these models can do.

Why do you think that? Why do you think a basic statistical continuation of the logic of a text wouldn't do what the current model does? There are trillions of conversations out there it can rely on to continue the text, people playing theatre, people roleplaying, tutorials, people playing opposite games, people brainstorming etc. Create a parser that can parse those down to logic, then make a markov chain based on that, and I have no problem seeing the current ChatGPT skills manifesting from that.

> Or alternatively, if you want to consider the model as a markov chain mapping the probability from the previous four thousand tokens to the next token then the space is astronomically large. Beyond astronomically and even economically large, there are ~50,000^4096 possible input states.

Yes, that is the novel thing, it compresses the states down to something manageable without losing the essence of the text, and then builds a model there of likely next token.

Post reply on HN