Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What is ChatGPT doing and why does it work?

writings.stephenwolfram.com

Post reply on HN
Print view View original

321–330 of 518 posts

Re: What is ChatGPT doing and why does it work?

#321

Post by fluoridation »

lelanthran wrote: post #219
cm2012 wrote: post #188

Earlier quoted context omitted.

For now. Give it a truly persistent memory and 100x the size of the dataset I think most people would change their tune.

> For now. Give it a truly persistent memory and 100x the size of the dataset I think most people would change their tune. Why does it need 100x the dataset? Sentient creatures, including humans, manage to figure stuff out from as little as a single datapoint . For a human to differentiate between a cat and a dog takes, maybe, two examples of each, not a few million pictures. An adult human who sees a hotdog for the…

>Why does it need 100x the dataset? Sentient creatures, including humans, manage to figure stuff out from as little as a single datapoint.

Human brains are not quite blank slates at birth. They're predisposed to interpret and quickly learn from the sort of inputs that their ancestors were exposed to. That is to say, the brain, which learns, is also the result of a learning process. If a mad scientist rewired your brain to your senses such that its inputs were completely scrambled and then deposited you on an alien planet, it might take your brain several lifetimes to restructure itself enough to interpret this novel input.

Re: What is ChatGPT doing and why does it work?

#322

Post by saberience »

space_fountain wrote: post #266
Salgat wrote: post #258

Earlier quoted context omitted.

I don't think you understand the scope of training data required for these models. We're talking thousands of lifetimes worth of reading for ChatGPT (GPT-3 for example is trained on 45TB of textual data).

I was responding to someone claiming humans learn these things with only one or two examples. I am aware of that GPT3 pretty much scraped every bit of text Open AI could find on the internet and I agree that probably makes it less example efficient than humans. But I also think this critique is slightly unfair, your brain has had the benefit of thousands of lifetimes of experience informing their structure and in bui…

The human brain hasn't had to "evolve" to learn writing. Our brain hasn't really changed for many thousands of years and writing has only been around for about 5000 years so we can't use the argument that "human brains have evolved over millions of years to do this" - it's not true.

GPT3 essentially needs millions of human years of data to be able to speak English correctly but still make obvious mistakes to us, so there's clearly something massive still missing.

Re: What is ChatGPT doing and why does it work?

#323

Post by sangnoir »

The number of people ITT who are reading interpreting ChatGPTs output as intelligence is too damn high.

I thought Ex Machina was unrealistic because of its dependence on AGI, or at least having a theory of mind. As it turns out, in the real world,a LLM trained on Tinder data could probably get the job done.

Re: What is ChatGPT doing and why does it work?

#324

Post by tiborsaas »

lelanthran wrote: post #219
cm2012 wrote: post #188

Earlier quoted context omitted.

For now. Give it a truly persistent memory and 100x the size of the dataset I think most people would change their tune.

> For now. Give it a truly persistent memory and 100x the size of the dataset I think most people would change their tune. Why does it need 100x the dataset? Sentient creatures, including humans, manage to figure stuff out from as little as a single datapoint . For a human to differentiate between a cat and a dog takes, maybe, two examples of each, not a few million pictures. An adult human who sees a hotdog for the…

> Why does it need 100x the dataset? Sentient creatures, including humans, manage to figure stuff out from as little as a single data point.

Because machine training doesn't involve embodiment and sensory information. Humans can extrapolate information from seeing a single image because we are "trained" from birth by being a physical actor in this world.

We know what bread looks like, what's mustard is, what a sausage is. We have information about size, texture, weight... all sorts of physical guesses that would help us pick the right tool for the job.

Machine training only relies on the coherent information we gave it to them, but that data also represents something we've created by experiencing the world through our bodies. So giving them more data can increase the model precision, I'd assume. It's also a kind of shortcut to intelligence, since we don't have to wait years/decades to make these models do some useful work.

Re: What is ChatGPT doing and why does it work?

#325

Post by aix1 »

pixl97 wrote: post #43
codeulike wrote: post #32

Wow this is 19,000 words. I like his summary at the end: At some level it’s a great example of the fundamental scientific fact that large numbers of simple computational elements can do remarkable and unexpected things. And this: ... But it’s amazing how human-like the results are. And as I’ve discussed, this suggests something that’s at least scientifically very important: that human language (and the patterns of th…

kurzgesagt has a new episode today about the human machine. https://www.youtube.com/watch?v=TYPFenJQciw One of the topics in the video was emergence, and where we see it in the world as more layers of complexity are added to systems. We're to the point with our algorithms that we are seeing complex system emerge from simple parts.

Does anyone know if there's a German version of this video? I tried finding it but couldn't.

Re: What is ChatGPT doing and why does it work?

#326

Post by sakex »

bheadmaster wrote: post #8
sakex wrote: post #6

Earlier quoted context omitted.

I don't re-read the whole sentence before every word I type, contrary to transformers. Also, I can go back and correct my mistakes.

> I don't re-read the whole sentence before every word I type Maybe you don't do it consciously, but your brain is quite aware of every word you typed before the word you're typing.

I would assume, more in a fashion close to LSTMs with a hidden state than by re-ingesting all the tokens every time.

Re: What is ChatGPT doing and why does it work?

#327

Post by hinkley »

rolenthedeep wrote: post #246
jeremyjh wrote: post #176

Earlier quoted context omitted.

Its not programming anymore its prompting. Prompting it to write and run the program that does what you want.

I think, eventually, this is where we end up. In not too many years, our job is going to be reviewing and debugging machine generated code. A few years after that, we're mostly caretakers and just keeping a human behind the wheel until we decide we don't need to watch the machines anymore. Things are unfortunately going to get much more interesting much sooner than people expect.

Don’t worry, the death of Dennard scaling and the specter of global warming will fix that, at least for some of us. There’s a lot of busy work and glue code to be automated but they’ve been trying to kill off development this way for at least forty years and all that changes is we get more sophisticated.

Re: What is ChatGPT doing and why does it work?

#328

Post by visarga »

biql wrote: post #282
lossolo wrote: post #79

Earlier quoted context omitted.

Intelligence exists without language, language is only a way to describe the world around us and transfer information. We personally can't experience intelligence without language because you know language all your life and can't remember a moment in which you didn't. But there were humans in history that didn't knew any language and they were intelligent, there are animals that do not know any language and are intel…

The remarkable thing is that language can combine intelligence of multiple beings and at some point that collective intelligence becomes a thing of it own, it can capture minds, spread and grow. Now we have all of our collective intelligence encoded in the form of billions of web pages and it seems that the language itself can direct its thinking, at least judging by the ChatGPT outputs (although, doesn't initiate it…

> can combine intelligence of multiple beings and at some point that collective intelligence becomes a thing of it own

Yes, that's what I meant. It is an evolutionary process with mutation and selection just like biology. We are just temporary hosts for these ideas that travel from person to person.

Re: What is ChatGPT doing and why does it work?

#329

Post by est »

GaggiX wrote: post #291
blackbear_ wrote: post #287

Earlier quoted context omitted.

> I failed to find the reason why ChatGPT worked as in contrast that gpt-3/2/1 never really worked (well) The answer may be as simple as "it has a lot more parameters", plus the additional fine-tuning from human conversation data.

GPT-3 has the same number of parameters as ChatGPT, it's just the finetuning (or alignment).

I kind of doubt it? https://arxiv.org/abs/2203.02155 it says

> outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters

Re: What is ChatGPT doing and why does it work?

#330

Post by ChaitanyaSai »

taejavu wrote: post #300
ChaitanyaSai wrote: post #293

Earlier quoted context omitted.

Fair point. I'd still argue that we are debating that it can 'merely' do a rap battle about consciousness is amazing. I think back to the time AlexNet came out. Would not have guessed this was possible in ten years. Also, with Eminem, I wonder how much of it having to skirt obscenities etc. As a tune and rhyme deaf person, what would you say is a good example of Eminem's rhyme scheme. Thanks!

This video is the clearest explanation of advanced rhyme patterns I’ve seen: https://youtu.be/QWveXdj6oZU

oh wow, this is perfect, thank you!
Post reply on HN