Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simply explained: How does GPT work?

confusedbit.dev

Post reply on HN
Print view View original

111–120 of 392 posts

Re: Simply explained: How does GPT work?

#111

Post by thanatropism »

masswerk wrote: post #57
seydor wrote: post #17

This is confusing, using the semantic vectors arithmetic of embeddings is not very relevant to transformers and its completely missing the word 'attention'. I don't think transformers are that difficult to explain to people , but it is hard to explain "why" they work. But i think it's important for everyone to look under the hood and know that there are no demons underneath.

> that there are no demons underneath Given that Lacan already proposed the unconscious as structured language-like more than half a century ago and described attention in his turn on Freud's impulse in favor of his concept of derive , we may say, this is pretty much where our own demons live. (I actually do think that revisiting Lacan in this context may be productive.)

We need to form some sort of guild of engineers who think Deleuze, Latour, Lacan et caterva should be read within our disciplines.

Re: Simply explained: How does GPT work?

#112

Post by anotherman554 »

ryanwaggoner wrote: post #78
Workaccount2 wrote: post #44

Earlier quoted context omitted.

> has burdened me with pseudo-philosophical questions about the nature of cognition that I am not well equipped to articulate, and make me wish I'd studied more neuroscience, philosophy, and comp sci earlier in life Welcome to the club. There pretty much are no answers, just theories primarily played out as thought experiments. Its on of those areas where you can pick out who knows less (or is being disingenuous) by…

It's interesting to me how many commenters on HN are absolutely convinced that GPT4 is incapable of thought or understanding or reasoning, it's "just" predicting the next word. And then they'll insist that it'll never be able to do things that it's already capable of doing... Interestingly, more than one of these folks have turned out to be religious. I wonder if increasingly intelligent AI systems will be challengin…

"Interestingly, more than one of these folks have turned out to be religious."

The guy fired by google for announcing LaMDA was sentient was religious.

I don't really see a meaningful distinction between declaring a machine is "thinking" for hand waving religious reasons and hand waving non-religious reasons, I'm afraid.

Re: Simply explained: How does GPT work?

#113

Post by ZeroGravitas »

simonw wrote: post #15
lm28469 wrote: post #11

Earlier quoted context omitted.

Care to post a full example ?

I used GPT-4 to build this tool https://image-to-jpeg.vercel.app using a few prompts the other day - my ChatGPT transcript for that is here: https://gist.github.com/simonw/66918b6cde1f87bf4fc883c677351...

I'm assuming the bits that say

> // Rest of the code remains the same

Are exactly as generated by GPT-4, i.e. it knew it didn't need to repeat the bits that hadn't changed, and knew to leave a comment like this to indicate that to the user.

It gets confusing when something can fake a human so well.

Re: Simply explained: How does GPT work?

#114

Post by samstave »

ZeroGravitas wrote: post #104

> It is able to link ideas logically, defend them, adapt to the context, roleplay, and (especially the latest GPT-4) avoid contradicting itself. Isn't this just responding to the context provided? Like if I say "Write a Limerick about cats eating rats" isn't it just generating words that will come after that context, and correctly guessing that they'll rhyme in a certain way? It's really cool that it can generate coh…

There once was a Cat in New York

Who got caught for feeding some Rats ; Tremendous Work!

All the people tell me, many men, biggly men - many with tears in their eyes...

That I have done nothing legally-wise

But the truth is ; I am an enormous dork.

>>_Created by an actual Human Being with actual DNA for crime scene evidence._

-

But just when they tried to brush under a rug

To try to make the folks 'shrug'

Is the Streisand Effect as a scar

As everyone knows of payments to a Porn Star

And the nation will know youre a simple thug.

Re: Simply explained: How does GPT work?

#115

Post by k__ »

z3c0 wrote: post #33
LeSaucy wrote: post #27

Earlier quoted context omitted.

I have only seen gpt generate imperative algorithms. Does it have the ability to work with concurrency and asynchrony?

I've attempted to pose a concurrency problem to GPT4. The output was invalid code, though likely would have looked correct to the untrained eye. It was only after I spelled out the limitations that it could account for them.

I tried point free solutions, which threw it off.

Re: Simply explained: How does GPT work?

#116

Post by nicpottier »

tel wrote: post #73
alkonaut wrote: post #41

What I wonder most is how it encodes knowledge/state other than in the sequence of queries/responses. Does it not have a "mind"? If I play a number guessing game, can I tell it to "think of a number between 0 and 100" and then tell me if the secret number is higher/lower than my guess (For a sequence of N guesses where it can concistently remember it's original number)? If not, why? Because it doesn't have context? I…

Yes, you absolutely can. I just did it and despite an initial warning that it couldn't play the game (always fun to see these) it absolutely did. The way it works is that each time it's tasked to produce a new response, it can view the entire history of the game. It knows that if it's said "higher" to 65 then it would be inconsistent to say "lower" to 64. Eventually this process terminates and the AI admits I "got" t…

Try making bad guesses and see it fall apart. (IE, guess something higher than your last guess when it says "too high") It is fully dependent on your part in the charade to keep the act up.

Re: Simply explained: How does GPT work?

#117

Post by myrmidon »

agentultra wrote: post #102
myrmidon wrote: post #92

Earlier quoted context omitted.

> We're talking about writing an emulator on a Harvard-architecture computer that can fully simulate the physics and biological processes the make up a human brain. By interpreting this system in our emulator we'd be able to witness a new human being that is indistinguishable from one that isn't simulated, right? My point is: if you don't believe that there is magic pixy dust in our brains, then this would NECESSARIL…

> I'm arguing that GPT4 is essentially the second approach Ok, so then it is an algorithm that simulates a specific behaviour that produces plausibly human-level results. My point is that this is not thinking , smart , or "general intelligence." Let's say I write an algorithm that can also produce text. It's not an implementation of the specification for GPT-4 but something novel. It takes the exact same inputs and p…

> My point is that this is not thinking, smart, or "general intelligence."

Why not? I would already, without hesitation, describe GPT4 as strictly more intelligent than my cat and also all gradeschoolers I've ever known... Maybe some adults, too- depends on your exact definition of intelligence.

> Let's say I write an algorithm [...], you can't tell if [input] was produced by GTP-4 or my algorithm.

Sure, I'd call your algorithm just as clever as GPT4 and approaching adult human levels of intelligence.

> It's still not intelligent, conscious, smart

Why not? What do these mean to you?

Re: Simply explained: How does GPT work?

#118

Post by EMM_386 »

seydor wrote: post #71
nitnelave wrote: post #50

Earlier quoted context omitted.

I was trying to keep the article at a level that everyone understands, from middle school up. I thought about going a bit deeper in the structure and mentioning attention, but my problem is that the intuitive concept of "attention" is quite different from the mathematical reality of an attention layer, and I'm sure I would have lost quite a few people there. It's always a trade-off :)

Indeed the word 'attention' implies some active process, which is also unfortunate, because it is really similarity maps over long ranges. This process apparenty creates a cross-reference maps of items that are commonly co-activated. But then what do the different heads do? Some kind of different "modalities of cross-reference"? I struggle to understand why this thing works the way it does. It's possible that Vaswani…

> I struggle to understand why this thing works the way it does.

I'm not in this field but have recently found myself going on the deepest dive possible into it as my small brain can absorb.

I now know about (on a surface level) neural networks, transformers, attention mechanisms, vectors, maticies, tokenization, loss functions and all sorts of other crazy stuff.

I come out of this realizing that there are some incredibly brilliant minds behind this. I knew AI was a complex subject but not on the level I've learned about now. To get what is essentially matrix multiplications to learn complex patterns and relationships in language is mind-blowing.

And it's creative. It can have a rap battle with an alter-ego, host a quiz party with other AIs of varying personalities, co-author a short story with me, respond to me only in emojis. The list is seemingly endless. Oh, and it can also do useful things. It's my programming companion too.

And we're just getting started.

Re: Simply explained: How does GPT work?

#119

Post by seydor »

dado3212 wrote: post #108
seydor wrote: post #17

This is confusing, using the semantic vectors arithmetic of embeddings is not very relevant to transformers and its completely missing the word 'attention'. I don't think transformers are that difficult to explain to people , but it is hard to explain "why" they work. But i think it's important for everyone to look under the hood and know that there are no demons underneath.

Is there an article that would meet your standard (discusses transformers, uses the word 'attention') that you could give to someone who has no familiarity with computer science?

Jay alammar's Illustrated transformer, although this too is detailed. I think it's still worth taking a look, because really i don't think that people have yet "compressed" what transformers do intuitively. None of the concepts of the networks are particularly hard math - it's basic algebra. But the overall construction is complicated.

https://jalammar.github.io/illustrated-transformer/

Re: Simply explained: How does GPT work?

#120

Post by ben_w »

myrmidon wrote: post #101
ben_w wrote: post #19

Earlier quoted context omitted.

To be deliberately unfair, imagine a huge if-else block — like, a few billion entries big — and each branch played out a carefully chosen and well-written string of text. It would convince a lot of people with the breadth, despite not really having much depth. The real GPT model is much deeper than that, of course, but my toy example should at least give a vibe for why even a simple thing might still feel extraordina…

This is absolutely not viable because exponential growth absolutely kills the concept. Such a system would already struggle with multiple-word inputs and it would be completely impossible to make it scale to even a paragraph of text, even if you had ALL of the observable universe at your disposal for encoding the entries. Consider: If you just have simple sentences consisting of 3 words (subject, object, verb, with 1…

α: most of those sentences are meaningless so they won't come up in normal use

β: if statements can grab patterns just fine in most languages, they're not limited to pure equality

γ: it's a thought experiment about how easy it can be to create illusions without real depth, and specifically not about making an AGI that stands up to scrutiny

Post reply on HN