Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What is ChatGPT doing and why does it work?

writings.stephenwolfram.com

Post reply on HN
Print view View original

291–300 of 518 posts

Re: What is ChatGPT doing and why does it work?

#291

Post by GaggiX »

blackbear_ wrote: post #287
est wrote: post #177

I've read many articles explains how gpt-3 work, and did a great job. What buffles me is the context consistency. ChatGPT was a huge leap compared to previous models. I have never seen it failed once. I often use "this" or "that" in my conversation with ChatGPT and it would guess 100% correct what I am refering to. Sometimes I paste a chunk of code and ask for questions of a specific part of it, ChatGPT fully underst…

> I failed to find the reason why ChatGPT worked as in contrast that gpt-3/2/1 never really worked (well) The answer may be as simple as "it has a lot more parameters", plus the additional fine-tuning from human conversation data.

GPT-3 has the same number of parameters as ChatGPT, it's just the finetuning (or alignment).

Re: What is ChatGPT doing and why does it work?

#293

Post by ChaitanyaSai »

taejavu wrote: post #286
ChaitanyaSai wrote: post #273

This misses that key point that all this prediction can give rise to what looks like astonishing human-level creativity and across many genres. The last decade and half have shown us that with enough data we can pick out patterns well enough to be able to "categorize". But to create , that seemed like a whole another human level outside the realm of mere prediction. Turns out it isn't. What exactly allows LLMs to hav…

I can’t answer your question but I would push back on the claim that it’s done the requested task very decently. There is nothing uniquely Eminem or Dennett about their respective parts. Eminem has never released a verse with as simplistic rhyme scheme as what’s been produced. Part of the mystique in your question is by assuming that’s it’s done the Eminem/Dennett part of the request justice, when it could really be…

Fair point. I'd still argue that we are debating that it can 'merely' do a rap battle about consciousness is amazing. I think back to the time AlexNet came out. Would not have guessed this was possible in ten years.

Also, with Eminem, I wonder how much of it having to skirt obscenities etc. As a tune and rhyme deaf person, what would you say is a good example of Eminem's rhyme scheme. Thanks!

Re: What is ChatGPT doing and why does it work?

#294

Post by visarga »

lukeplato wrote: post #65
visarga wrote: post #53

Earlier quoted context omitted.

> 'with enough data its easier to model than we expected' > a self-similar process is ultimately responsible for granting us our own intelligence In my view, intelligence essentially resides within language, specifically in the corpus of language. Both humans and AIs can be effectively colonized by language, as there are innumerable concepts and observations that are transmitted from one mind to another, and now even…

Yann LeCun often argues that animals like cats and dogs are substantially more intelligent than current LLMs [0] and I'd have to agree. I don't see how/why to consider practical knowledge as only constituting a tiny fraction of our overall intelligence. Either way, it's not clear if the GPT-* models will someday produce emergent common sense or if they're going down an entirely wrong path. [0] https://twitter.com/yle…

> I don't see how/why to consider practical knowledge as only constituting a tiny fraction of our overall intelligence.

A human without language would be just a less adapted ape. The difference comes from language (where I include culture, science and technology).

Today you have to be a PhD to push forward the boundaries of human knowledge, and only in a very narrow field. This is the amount we add back to culture - if you got one good original idea in your hole life you consider yourself lucky.

https://www.wasyresearch.com/content/images/2021/08/the_esse...

Re: What is ChatGPT doing and why does it work?

#295

Post by kortilla »

basch wrote: post #187
visarga wrote: post #53

Earlier quoted context omitted.

> 'with enough data its easier to model than we expected' > a self-similar process is ultimately responsible for granting us our own intelligence In my view, intelligence essentially resides within language, specifically in the corpus of language. Both humans and AIs can be effectively colonized by language, as there are innumerable concepts and observations that are transmitted from one mind to another, and now even…

Humans are somewhat a blank slate, and culture is our initial prompt. The variety of humans is because of the variety of our initial prompts, and our similarities are because of the similar characteristics of our various cultures. (I of course recognize some of our intelligence is genetic or epigenetic or microbial.)

No, chatgpt is already trained once it’s ready for prompts.

Re: What is ChatGPT doing and why does it work?

#296

Post by somenameforme »

ChaitanyaSai wrote: post #273

This misses that key point that all this prediction can give rise to what looks like astonishing human-level creativity and across many genres. The last decade and half have shown us that with enough data we can pick out patterns well enough to be able to "categorize". But to create , that seemed like a whole another human level outside the realm of mere prediction. Turns out it isn't. What exactly allows LLMs to hav…

I'd argue that the reason you're probably impressed with this is because it's outside your domain. You are probably less than impressed that it can generate entire code segments, or fluently answers arbitrary questions because it's easy to see how a training on Stack Exchange and other such sources can easily generate such output. By contrast somebody into rap, but with little knowledge of programming, would probably be floored at the LLM 'being a programmer.'

Now go search for 'rap battle example' or other such searches and find the web has even more such content, but you've probably never once in your life seen even a single example of it. And in fact it's also likely been trained on the entire history of every single song, rap, poem, etc. So it's just doing the same thing, but in a field outside your knowledge.

Re: What is ChatGPT doing and why does it work?

#297

Post by tempodox »

G_z9 wrote: post #249
osswid wrote: post #77

In the days when Sussman was a novice Minsky once came to him as he sat hacking at the PDP-6. "What are you doing?", asked Minsky. "I am training a randomly wired neural net to play Tic-Tac-Toe." "Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play" Minsky shut his eyes, "Why do you close your eyes?", Sussman asked his teacher. "So that the room will be empty." A…

This allegory has haunted me for ten years. I always see it posted here and invariably the poster doesn’t elaborate or explain at all. Im pretty sure the people who post this just do it to look smart. I can’t find an explanation anywhere on the web. I asked chatgtp what it means and it said this: This allegory, often referred to as an "AI koan," is a story that conveys a deeper meaning about the nature of artificial…

The way the net is wired is the preconception. Just as closing your eyes doesn't actually empty the room, randomizing the preconception doesn't make it go away.

Re: What is ChatGPT doing and why does it work?

#299

Post by theGnuMe »

kilgnad wrote: post #252
vore wrote: post #174

Earlier quoted context omitted.

They really aren't more than just statistical word generators because we can and do build what are exactly statistical word generators and get the behavior of ChatGPT (because that is what ChatGPT is). For what it's worth, this isn't a knock on ChatGPT, but more just how amazing how far you can get with straightforward concepts.

Except the paper in the OPs comment literally talks about how these LLM's are building actual models of the real world. Literally. The root parent post links a research paper that categorically falsifies your statement. No need to be worried about knocking chatGPT. I have no pride invested in the thing. But I do think that people who view it solely as statistical word generators are biased.

You can imagine that the embedding for the token “boat” includes wood, steel, water and so forth in some set of probabilistic paths as learnt by autorgressive training since the words appear together in the past N tokens. So they are directly in frame. A question is how to connect out of frame elements and are overlapping tokens sufficient to do “that”. Specifically is the token “that” sufficiently trained to reveal what it refers to? I think this depends on the fine tuning q/a task which adds in a layer of alignment rather than being an emergent property of the LLM in general.

Still alignment tasks are autoregressive (I think)… they could be masked or masked part of speech potentially.. but if autoregressive then I suspect you’re looking at regularities in positioning to identify things.

Re: What is ChatGPT doing and why does it work?

#300

Post by taejavu »

ChaitanyaSai wrote: post #293
taejavu wrote: post #286

Earlier quoted context omitted.

I can’t answer your question but I would push back on the claim that it’s done the requested task very decently. There is nothing uniquely Eminem or Dennett about their respective parts. Eminem has never released a verse with as simplistic rhyme scheme as what’s been produced. Part of the mystique in your question is by assuming that’s it’s done the Eminem/Dennett part of the request justice, when it could really be…

Fair point. I'd still argue that we are debating that it can 'merely' do a rap battle about consciousness is amazing. I think back to the time AlexNet came out. Would not have guessed this was possible in ten years. Also, with Eminem, I wonder how much of it having to skirt obscenities etc. As a tune and rhyme deaf person, what would you say is a good example of Eminem's rhyme scheme. Thanks!

This video is the clearest explanation of advanced rhyme patterns I’ve seen: https://youtu.be/QWveXdj6oZU
Post reply on HN