Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What is ChatGPT doing and why does it work?

writings.stephenwolfram.com

Post reply on HN
Print view View original

131–140 of 518 posts

Re: What is ChatGPT doing and why does it work?

#131

Post by inasio »

scottlawson wrote: post #112

I spent about an hour reading this. I know most people probably won't be able to do the same because of time constraints in their day, but seriously this is the most awesome post about large language models I've read. Take a second to think about how much time it must have taken to write this, to prepare all of these examples. This post gave me another way to look at something I already partially understood pieces of…

I'm typically a bit wary of Wolfram's posts, but it is very often the case that the content is good enough that I don't mind a bit of self-aggrandizing. This one though I found to be just pure great, super easy and clear intro into the "magic" of LLMs. Thank you.

Re: What is ChatGPT doing and why does it work?

#132

Post by britzkopf »

codeulike wrote: post #32

Wow this is 19,000 words. I like his summary at the end: At some level it’s a great example of the fundamental scientific fact that large numbers of simple computational elements can do remarkable and unexpected things. And this: ... But it’s amazing how human-like the results are. And as I’ve discussed, this suggests something that’s at least scientifically very important: that human language (and the patterns of th…

The thing I'm sort of confused about, but maybe someone can explain why I shouldn't be, is, why does there seem to be no implication for language translation? Or is there but coverage is overwhelmed by the fascination with chatGPT? In short, is machine language translation now a fully solved problem? A couple years ago when I tested Google translate in a non-esoteric conversation with my Russian speaking girlfriend and, although it was useful, in terms of native fluency it failed pretty decisively. But isn't this a much easier problem than that which chatGPT is being marketed (or at least covered in the media) as solving?

Re: What is ChatGPT doing and why does it work?

#133

Post by nestorD »

everyone wrote: post #114

I am a game programmer but in my spare time I like to learn and get experience in random interesting areas. Eg. recently I learned electronics and Arduinos. Would ye recommend any projects I could do in order to get experience with and learn about this new AI stuff like ChatGPT?

Take a look at langchain. It lets you interface with language models and has a number of interesting demos to get you started.

Re: What is ChatGPT doing and why does it work?

#134

Post by vore »

kilgnad wrote: post #129
spion wrote: post #15

The answer to this is: "we don't really know as its a very complex function automatically discovered by means of slow gradient descent, and we're still finding out" Here are some of the fun things we've found out so far: - GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (…

Finally I'm tired of people saying it's just a probabilistic word generator and downplaying everything as if they know. If you said something along these lines before... then these papers show that you're not fully grasping the situation here. There are clearly different angles of interpreting what these models are actually doing but people are stubbornly refusing to believe it's anything more then just statistical w…

Doesn't the article say... exactly that?

  The basic concept of ChatGPT is at some level rather simple. Start from a huge sample of human-created text from the web, books, etc. Then train a neural net to generate text that’s “like this”. And in particular, make it able to start from a “prompt” and then continue with text that’s “like what it’s been trained with”.
Just because there are emergent behaviors doesn't mean it's not a probabilistic word generator. Nor does it being a probabilistic word generator mean it can't have interesting underlying properties.

Re: What is ChatGPT doing and why does it work?

#135

Post by kilgnad »

vore wrote: post #134
kilgnad wrote: post #129

Earlier quoted context omitted.

Finally I'm tired of people saying it's just a probabilistic word generator and downplaying everything as if they know. If you said something along these lines before... then these papers show that you're not fully grasping the situation here. There are clearly different angles of interpreting what these models are actually doing but people are stubbornly refusing to believe it's anything more then just statistical w…

Doesn't the article say... exactly that? The basic concept of ChatGPT is at some level rather simple. Start from a huge sample of human-created text from the web, books, etc. Then train a neural net to generate text that’s “like this”. And in particular, make it able to start from a “prompt” and then continue with text that’s “like what it’s been trained with”. Just because there are emergent behaviors doesn't mean i…

I didn't say it's not a probabilistic word generator. I'll quote more of what I wrote:

>There are clearly different angles of interpreting what these models are actually doing but people are stubbornly refusing to believe it's anything more then just statistical word jumbles.

So CLEARLY because I said there are different angles of interpretation I'm implying that from one of these angles we can interpret it as a statistical word generator.

I mean from one perspective you and I are both statistical word generators too.

Re: What is ChatGPT doing and why does it work?

#136

Post by lossolo »

sigmoid10 wrote: post #84
lossolo wrote: post #76

Earlier quoted context omitted.

> "E3, D3, C4 ..." and NOTHING else. It knows nothing of the board. It doesnt event know that there are two players. Yeah, like language have gramma rules games also have rules, in both cases LLM can learn rules, it's the same with many other structured chains of actions/tokens, you could also model actions from different domains and use them as language. It seems a lot of emergent behaviours of LLMs are what you cou…

Knowledge distillation for transformers is already a thing and it is still actively researched since the potential benefits of not having to run these gigantic models are enormous.

Indeed but all the papers I've read about this were only for simple transformers, for now at least.

Re: What is ChatGPT doing and why does it work?

#137

Post by thomashop »

britzkopf wrote: post #132
codeulike wrote: post #32

Wow this is 19,000 words. I like his summary at the end: At some level it’s a great example of the fundamental scientific fact that large numbers of simple computational elements can do remarkable and unexpected things. And this: ... But it’s amazing how human-like the results are. And as I’ve discussed, this suggests something that’s at least scientifically very important: that human language (and the patterns of th…

The thing I'm sort of confused about, but maybe someone can explain why I shouldn't be, is, why does there seem to be no implication for language translation? Or is there but coverage is overwhelmed by the fascination with chatGPT? In short, is machine language translation now a fully solved problem? A couple years ago when I tested Google translate in a non-esoteric conversation with my Russian speaking girlfriend a…

I think general translation is kind of solved when it comes to popular languages. Try DeepL.

I dont know how well it works for different language pairs to the languagesi know. I dont even know if deepl uses one of the newer large language models

Re: What is ChatGPT doing and why does it work?

#138

Post by busyant »

ian0 wrote: post #113
rubyist5eva wrote: post #68

It's a bunch of smoke and mirrors akin to blockchain, bitcoin (and the entire web3/crypto space in general) - a whole bunch of hype and "evangelists" who keep saying it's going to change the world (and make them a whole lot of money in the process, how convenient!) but if you do look at it critically even at a surface level you realize it's just a bunch of really computationally expensive BS that isn't any better or…

Its not, really. The major difference between this and web3/crypto is utility. Its actually useful. In our office we have people from three different teams using it near daily. Out of choice. As do many kids in our network for homework. For a recently released product effectively in beta thats insane. Theres a bunch of snake oil salesmen jumping on the bandwagon which is very unfortunate. But lots of people sell fake…

> Its actually useful. In our office we have people from three different teams using it near daily. Out of choice. As do many kids in our network for homework. For a recently released product effectively in beta thats insane.

Yep.

I asked it to make a worksheet for students to practice converting numbers written in scientific notation back to "standard" format.

So, it gave me a bunch of output like:

6.2x10^-6: ___________________

What annoyed me about this is that it used the letter "x" instead of the proper multiplication symbol "×" and it used the hyphen (-) instead of the appropriate "minus" sign (−).

So, I told it to use proper typographic symbols, and it did!

It converted "6.2x10^-6" to "6.2×10^−6"

It even told me the Unicode numbers it was using for × and −.

Then I asked it to re-generate the worksheet using LaTeX and the siunitx package.

It nailed it.

It's like someone just handed me a turbo-charged assistant. Yeah, I have to make sure my assistant hasn't gone insane, but it has already spared me a ton of grunt work.

Re: What is ChatGPT doing and why does it work?

#139

Post by vore »

kilgnad wrote: post #135
vore wrote: post #134

Earlier quoted context omitted.

Doesn't the article say... exactly that? The basic concept of ChatGPT is at some level rather simple. Start from a huge sample of human-created text from the web, books, etc. Then train a neural net to generate text that’s “like this”. And in particular, make it able to start from a “prompt” and then continue with text that’s “like what it’s been trained with”. Just because there are emergent behaviors doesn't mean i…

I didn't say it's not a probabilistic word generator. I'll quote more of what I wrote: >There are clearly different angles of interpreting what these models are actually doing but people are stubbornly refusing to believe it's anything more then just statistical word jumbles. So CLEARLY because I said there are different angles of interpretation I'm implying that from one of these angles we can interpret it as a stat…

I mean, that is the fundamental architecture of the model and how it's explicitly designed: given an input embedding and output embedding, the model is trained over a massive corpus of text to optimize the probability for generating the next token in the sequence. Any other interpretation is secondary to that, no matter how impressive the results are.

Re: What is ChatGPT doing and why does it work?

#140

Post by everyone »

cypress66 wrote: post #125
everyone wrote: post #114

I am a game programmer but in my spare time I like to learn and get experience in random interesting areas. Eg. recently I learned electronics and Arduinos. Would ye recommend any projects I could do in order to get experience with and learn about this new AI stuff like ChatGPT?

Andrej Karpathy videos

Tx but I dont wanna watch videos. I want to build something.
Post reply on HN