Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT Explained: A normie's guide to how it works

jonstokes.com

Post reply on HN
Print view View original

61–70 of 144 posts

Re: ChatGPT Explained: A normie's guide to how it works

#61

Post by oblib »

oblib wrote: post #3

I posted some comments about ChatGTP in a local FB group and there was a pretty large percent of folks who responded here that think it's just an awful thing that's going to lead to the downfall of civilization. I tried to offer that it is pretty cool, but it's just software that basically presents search engine results in a different manner along with a few other tricks, but it's not "HAL". I live in a very red and…

Wow... I didn't expect to get so many panties knotted up with that observation, but I think those who down voted it prove my observation.

What bugs me most about ChatGPT is that it doesn't attribute where it got the results it offers. For example, I asked it: "how to save a document with PouchDB" and it showed me the code, and while I didn't compare it, it looks like a copy and paste from PouchDB.com, and that should be referenced with a link in the response if that's the case, but it was not.

Re: ChatGPT Explained: A normie's guide to how it works

#62

Post by iamflimflam1 »

I’m not quite sure this article is really going to help “normies”. It seems to fall into the classic trap of “I’m going to explain something in simple terms that you will understand. But, I still want you to think I’m really clever so I’m going to make sure you still don’t really understand”

Re: ChatGPT Explained: A normie's guide to how it works

#63

Post by hgsgm »

starik36 wrote: post #21

ChatGPT is probably the first software product that I have no idea how I'd go about implementing. I watched a number of YT videos about it including Andrew Carpathy's 2 hour coding session building a mini GPT. I understand the process abstractly, but I am unable to grok the details about how it's able to take my vague interpretation of what I want and then write code and actually give me exactly what I wanted.

Do you know how you'd go about implementing a video codec? A Haskell compiler? Linux kernel? Efficient relational database? wifi 802.11n radio? An efficient classical AI chess engine?

Re: ChatGPT Explained: A normie's guide to how it works

#64

Post by LakshyAAAgrawal »

spion wrote: post #54
ChatGTP wrote: post #30

Earlier quoted context omitted.

So basically, if a pattern that hasn't been encountered before is seen, it will just try to connect "something" together, which is why it does things like predict today's date being in the future etc? The model says, "I don't have a good enough path forwards here, I'll just make one up given the next best thing I have and serve it back"? Maybe this is why Bing is working differently, they've changed the model or the…

Not just connect something. For example, at a certain size, GPT models start to "learn" how to do basic arithmetic (addition), even for numbers with multiple digits they've never encountered before. It might look like a small thing, what with computers being quite able to do arithmetic at the base level. But this is a language model, so its a bit different. It learns how to add numbers without "carrying the 1" first,…

Hey! That's a very interesting explanation, could you please provide any references for further/detailed reading on model's abilities to learn to add numbers, with carry and finally with carry across digits?

Re: ChatGPT Explained: A normie's guide to how it works

#65

Post by ChatGTP »

spion wrote: post #54
ChatGTP wrote: post #30

Earlier quoted context omitted.

So basically, if a pattern that hasn't been encountered before is seen, it will just try to connect "something" together, which is why it does things like predict today's date being in the future etc? The model says, "I don't have a good enough path forwards here, I'll just make one up given the next best thing I have and serve it back"? Maybe this is why Bing is working differently, they've changed the model or the…

Not just connect something. For example, at a certain size, GPT models start to "learn" how to do basic arithmetic (addition), even for numbers with multiple digits they've never encountered before. It might look like a small thing, what with computers being quite able to do arithmetic at the base level. But this is a language model, so its a bit different. It learns how to add numbers without "carrying the 1" first,…

The only issue with the math example would be that we don't do complex math in English. Of course it can learn other "languages" but as you said there is less "written" complex math

It probably also contains true generalizations that humanity hasn't thought of before, as well as specializations of those generalizations that could be immensely useful.

I thought this is quite exciting, I actually think this will become a field of research in itself, and that might spur completely novel new industries and field of technology, research etc.

I'm being optimistic here but I hope we're on the cusp of a greater future, rather than a dystopian one brought on by terminators and AI that replaces all work :)

Re: ChatGPT Explained: A normie's guide to how it works

#66

Post by elif »

justusthane wrote: post #50
notahacker wrote: post #29

Earlier quoted context omitted.

Technically it's a transformation rather than a translation because Pig Latin isn't a real language, it's the product of a word game to make up new words out of English words by rearranging them according to certain rules. The article does assume you know this though (and doesn't consider the alternative of picking a simpler example like cat to CAT in capital letters or cat to chat in French) Same goes for its super…

“Translate: to transfer or turn from one set of symbols into another” – Merriam-Webster No, “translate” still works here, regardless of Pig Latin not being a language. And I agree with the GP that it’s more intuitively understandable word than “transform.”

You can call it both a transformation or a translation, but that doesn't make 'translation' more appropriate of a term. It is a procedural transformation that can be expressed entirely in a one line function of pseudo-code.

translation from one real language to another is so complex that it cannot even be represented in a hash table. Notably, the elements of grammar and syntax are not present in this naive form of translation. It is also lacking the element of bridging understanding, the root of the word translate.

I would even posit that many CS words that are more appropriate here: encoding, compiling, cyphering, mutating, obfuscating,

Thus insisting on a word where critical elements of that word are lacking and despite the existence of more precise language is itself a poor translation.

Re: ChatGPT Explained: A normie's guide to how it works

#67

Post by hprotagonist »

ChatGTP wrote: post #65
spion wrote: post #54

Earlier quoted context omitted.

Not just connect something. For example, at a certain size, GPT models start to "learn" how to do basic arithmetic (addition), even for numbers with multiple digits they've never encountered before. It might look like a small thing, what with computers being quite able to do arithmetic at the base level. But this is a language model, so its a bit different. It learns how to add numbers without "carrying the 1" first,…

The only issue with the math example would be that we don't do complex math in English. Of course it can learn other "languages" but as you said there is less "written" complex math It probably also contains true generalizations that humanity hasn't thought of before, as well as specializations of those generalizations that could be immensely useful. I thought this is quite exciting, I actually think this will become…

Borges, of course, kind of skewered your hopes about 80 years ago.

Re: ChatGPT Explained: A normie's guide to how it works

#68

Post by phire »

spion wrote: post #54
ChatGTP wrote: post #30

Earlier quoted context omitted.

So basically, if a pattern that hasn't been encountered before is seen, it will just try to connect "something" together, which is why it does things like predict today's date being in the future etc? The model says, "I don't have a good enough path forwards here, I'll just make one up given the next best thing I have and serve it back"? Maybe this is why Bing is working differently, they've changed the model or the…

Not just connect something. For example, at a certain size, GPT models start to "learn" how to do basic arithmetic (addition), even for numbers with multiple digits they've never encountered before. It might look like a small thing, what with computers being quite able to do arithmetic at the base level. But this is a language model, so its a bit different. It learns how to add numbers without "carrying the 1" first,…

Worth pointing out that it's not 100% accurate at arithmetic, that becomes quite clear when you ask it to calculate large numbers. It's result is usually close, but not perfect.

So it hasn't learned the exact rules for arithmetic, it's learned rules for approximating arithmetic to a decent level of accuracy. Similar to how humans can know the approximate result of an equation before doing the actual math (though GPT is way more precise than humans)

Re: ChatGPT Explained: A normie's guide to how it works

#69

Post by jimbokun »

spion wrote: post #5

I like the token window bit. I don't really like the probability bit, because it kinda alludes that OpenAI just built a huge probability map of all N-grams (N=8000) and called it a day. Which incidentally would also imply that a lot of N-grams just don't exist in the training data, causing the model to completely halt when someone says something unexpected. But that's not the case - instead we convert words into a lo…

The author knows all that, he’s just trying to simplify to give the layman some intuition about what’s going on.
Post reply on HN