Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simply explained: How does GPT work?

confusedbit.dev

Post reply on HN
Print view View original

271–280 of 392 posts

Re: Simply explained: How does GPT work?

#271

Post by pegasus »

lordnacho wrote: post #246
dvt wrote: post #223

Earlier quoted context omitted.

> It seems to me to posit that to understand requires that the understandee is human. Here's a thought experiment. Suppose we make first contact tomorrow, and we meet some intelligent aliens. What are some questions you would ask them? How would you decide on their sentience or understanding? Sentience involves goal-seeking, understanding, sensory inputs, first-personal mental states (things like pain, happiness, sad…

Does that mean sentient beings need to have had the evolutionary history that people have had? After all why would aliens have feelings like we have feelings? Don't we have feelings because either it was useful to our ancestors, or they are a spandrel of something that was?

Maybe that's all there is to it. But maybe not. These questions are not settled. It might seem obvious to you, but it doesn't seem so to me. Check out some of my other comments in this thread if you'd like to understand where my reservations come from.

Re: Simply explained: How does GPT work?

#272

Post by jacksnipe »

i-use-nixos-btw wrote: post #130

I’d be interested in hearing from anyone who takes the Chinese Room scenario seriously, or at least can see how it applies to any of this. I cannot see that it matters if a computer understands something. If it quacks like a duck and walks like a duck, and your only need is for it to quack and walk like a duck, then it doesn’t matter if it’s actually a duck or not for all intents and purposes. It only matters if you…

This is also my interpretation of the Chinese Room.

Norvig and Chomsky really got into this type of argument, though maybe it’s a stretch to say it’s this exact one; see Norvig’s side here: https://norvig.com/chomsky.html

Re: Simply explained: How does GPT work?

#273

Post by JieJie »

brazzy wrote: post #192
JieJie wrote: post #132

Earlier quoted context omitted.

I literally lost a friend of thirty years yesterday because she is wedded to the Chinese Room analogy so fiercely, she refuses to engage on the subject at all. For all the terrible things people worry about ChatGPT doing, this was not one that I thought I was going to have to deal with. (edit: ChatGPT was not involved at all, but when I suggested she give it a try to see for herself, that was the end of it.)

How can an disagreement over GPT possibly be something to end a friendship over? This sounds like you said "I cannot possibly be friends with someone who does not believe that LLMs are emerging AGI!", and people read it like that and are downvoting you. I'm gonna assume the situation was more complex, but still find it hard to imagine, how a disagreement over such an academic topic could end up destroying a friendshi…

It was a complex situation to be sure.

I only shared the story to illustrate how personally people are taking these discussions. I really felt like I was being very neutral and just sharing my enthusiasm. It was entirely unwelcome, apparently.

If there's a lesson to be learned it's that people's tempers over these issues may be hotter than they appear.

I can barely speak with my artist friends on the issue these days due to their generative AI fears. Their emotions are completely intractable on the subject: AI art is theft. Period.

Re: Simply explained: How does GPT work?

#274

Post by int_19h »

cgearhart wrote: post #105
onetrickwolf wrote: post #7

I've been using GPT4 to code and these explanations are somewhat unsatisfactory. I have seen it seemingly come up with novel solutions in a way that I can't describe in any other way than it is thinking. It's really difficult for me to imagine how such a seemingly simple predictive algorithm could lead to such complex solutions. I'm not sure even the people building these models really grasp it either.

It’s a fallacy to describe what the machine does as “thinking” because that’s only process you know for achieving the same outcome. When you initiate the model with some input where you expect some particular correct output, that means there exists some completed sequence of tokens that is correct—if that weren’t true then you either wouldn’t ask or else you wouldn’t blame the model for being wrong. Now imagine a mac…

Now consider the case when you tell GPT to "think it out loud" before giving you the answer - which, coincidentally, is a well-known trick that tends to significantly improve its ability to produce good results. Is that thinking?

Re: Simply explained: How does GPT work?

#275

Post by danenania »

squeaky-clean wrote: post #250
danenania wrote: post #123

Earlier quoted context omitted.

It's all about emergent complexity. While you can reduce it to "just" statistical auto-completion of the next word, we are seeing evidence of abstraction and reasoning produced as a higher-order effect of these simple completions. It's a bit like the Sagan quote: "If you wish to make an apple pie from scratch, you must first invent the universe". Sometimes for GPT to "just" complete the next word in a way that humans…

Turns out good usage of "language" requires a model of the world in which that language exists. "The purple, two eyed, green, five eyed, invisible frog said moo" is a grammatically fine sentence. But logically it makes no sense, does it have two eyes or five? Is it green or purple or invisible? Frogs don't typically say moo. To have actual coherent usage of language, you need a model of the world. Not just the world,…

Very well said. We think of a word as "just" a word, a simple, primarily textual thing, but it's actually a vertex on an extremely large and complex many-dimensional graph that includes connections related to meaning, logic/reasoning, knowledge about reality, emotional sentiment, and so on. The literal textual representation of the word--the letters it consists of--are just one property among many, and probably one of the least important to producing sensible output. GPT is discovering the shape of this super-graph and learning to navigate its connections.

Re: Simply explained: How does GPT work?

#276

Post by int_19h »

lm28469 wrote: post #103
simonw wrote: post #15

Earlier quoted context omitted.

I used GPT-4 to build this tool https://image-to-jpeg.vercel.app using a few prompts the other day - my ChatGPT transcript for that is here: https://gist.github.com/simonw/66918b6cde1f87bf4fc883c677351...

See my problem with virtually every single example is that we talk about "I can't describe in any other way than it is thinking", "such complex solutions" but in the end we get a 50 lines "app" that you'd see in a computer science 101 class It's very nice, it's very impressive, it will help people, but it doesn't align with the "you're just about to lose your job" "Skynet comes in the next 6 months" &c. If these basi…

The concern is the velocity. GPT-4 can solve tasks today that it couldn't solve one months ago. And even one month ago, the things it could do made GPT-3.5 look like a silly toy.

Then there's the question of how much this can be scaled further simply by throwing more hardware at it to run larger models. We're not anywhere near the limit of that yet.

Re: Simply explained: How does GPT work?

#277

Post by int_19h »

ZeroGravitas wrote: post #113
simonw wrote: post #15

Earlier quoted context omitted.

I used GPT-4 to build this tool https://image-to-jpeg.vercel.app using a few prompts the other day - my ChatGPT transcript for that is here: https://gist.github.com/simonw/66918b6cde1f87bf4fc883c677351...

I'm assuming the bits that say > // Rest of the code remains the same Are exactly as generated by GPT-4, i.e. it knew it didn't need to repeat the bits that hadn't changed, and knew to leave a comment like this to indicate that to the user. It gets confusing when something can fake a human so well.

Yes, it will do that routinely. For example, you can ask it to generate HTML/JS/SVG in a single file to render some animated scene, and then iterate on that by telling it what looks wrong or what behaviors you like to change - and it will answer by saying things like, "replace the contents of the element with the following".

Re: Simply explained: How does GPT work?

#278

Post by int_19h »

agentultra wrote: post #142
myrmidon wrote: post #117

Earlier quoted context omitted.

> My point is that this is not thinking, smart, or "general intelligence." Why not? I would already, without hesitation, describe GPT4 as strictly more intelligent than my cat and also all gradeschoolers I've ever known... Maybe some adults, too- depends on your exact definition of intelligence. > Let's say I write an algorithm [...], you can't tell if [input] was produced by GTP-4 or my algorithm. Sure, I'd call you…

> I would already, without hesitation, describe GPT4 as strictly more intelligent than my cat Well if we're going to define intelligence based one what you believe it is then why don't you explain it? I'm not the one claiming to know what intelligence is or that we can even simulate a system capable of emulating this characteristic. So if you hold the specification for human thought I think you ought to share it with…

You're the one claiming that GPT is not in any sense, shape, or form intelligent. Such claim inevitably carries a very strong implication that you know what intelligence is.

Re: Simply explained: How does GPT work?

#279

Post by int_19h »

vadansky wrote: post #31
habosa wrote: post #20

Is it possible that we don’t truly know how it works? That there is some emergent behavior inside these models that we’ve created but not yet properly described? I’ve read a few of these articles but I’m still not completely satisfied.

I hate being the bearish guy during the hype cycle, but I think a lot of that is just anthropomorphizing it. They fed it TBs of human text, it spits out human text, we think it's humanesque. Of course maybe I'm wrong and it's AGI and it will find this comment and torture me for for insulting it's intelligence.

It doesn't just "spit out text". You can give it instructions, and it carries them out. If you give it access to an API, it can actually use that API to perform tasks (e.g. draw images). And it can do all that for tasks that are verifiably not a part of its training set.
Post reply on HN