Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is ChatGPT Doing and Why Does It Work? (2023)

writings.stephenwolfram.com

Post reply on HN
Print view View original

91–100 of 100 posts

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#91

Post by danielmarkbruce »

mg wrote: post #43
faizshah wrote: post #22

Earlier quoted context omitted.

This is your human interpretation of what it is doing. Another interpretation could be that it has seen similarly structured sentences in other logic/lateral thinking problems so it raises the likelihood of words that have to do with logical inconsistencies and then out of that vocabulary the highest likelihood words are those that come from the nearby context. I would posit that this interpretation is significantly…

You mean a prompt in the form of "A happened, then B happened, then C happened" activates the word "inconsistency"? We can test that by slightly altering the text so that there is no inconsistency anymore and see if the model still claims there is one. We can do so by adding the word "mother" after the word "her": What do you think about the following text? Joe drove Sue to university. Afterwards he drove home again…

They are training on 10's of millions of tokens/words and representing each in high dimensional space. In practice, it's seen a lot of stuff.

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#92

Post by misnome »

williamcotton wrote: post #84
Jensson wrote: post #77

Earlier quoted context omitted.

I just tried with 4o and it didn't notice anything, it just rewrote the statements with different words.

https://news.ycombinator.com/item?id=40721115

You explicitly asked it to explain?

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#93

Post by namaria »

danielmarkbruce wrote: post #89
namaria wrote: post #81

Earlier quoted context omitted.

Claiming LLMs are manipulating 'high dimension concepts' not words when all they can do with these 'high dimension concepts' is outputting words is meaningless.

It's meaningful because the model can learn relationships between things (like, that a dog and cat are often pets and interchangdable in certain circumstances) and then produce output that it has never seen. Read the word2vec paper as an example. And, an LLM can output whatever you want. Change the last linear layer from (embedding dimension -> vocab size) to (embedding dimension -> sentiment categories) for example,…

This isn't magic. Precisely. Calling it manipulating high dimension concepts muddles the waters. It's not. It's merely using a whole lot of numbers to manipulate word tokens.

And it's quite limited at that despite the impressive ability to store up language corner cases and put them back out.

"an LLM can output whatever you want" -> this right here is approaching magical thinking. I am really clear headed about it's capabilities and limitations, breathlessly describing it's internals as if it implies it's anything more then a first decent NLP interface is the problem and you seem to be indulging in it.

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#94

Post by namaria »

mensetmanusman wrote: post #87
namaria wrote: post #82

Earlier quoted context omitted.

Small models can't do impressive language corner cases. And the benefits of the models that can scale linearly while their cost grows geometrically. Also claiming training costs are 'one off' when there's so much resources being poured into training new and bigger models is disingenuous.

The resources pouring in still pale in comparison to capital expenditures in the semi conductor industry.

The inflation in semi conductor capital expenditure is the problem caused by the AI hype.

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#95

Post by williamcotton »

misnome wrote: post #92
williamcotton wrote: post #84

Earlier quoted context omitted.

https://news.ycombinator.com/item?id=40721115

You explicitly asked it to explain?

You need to ask it something or else it is just a weird sentence floating in a void. What would a reasonable response be? “And..?”

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#96

Post by mensetmanusman »

namaria wrote: post #94
mensetmanusman wrote: post #87

Earlier quoted context omitted.

The resources pouring in still pale in comparison to capital expenditures in the semi conductor industry.

The inflation in semi conductor capital expenditure is the problem caused by the AI hype.

Nah, tsmc spending tens of billion annually on capital before OpenAI.

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#97

Post by danielmarkbruce »

namaria wrote: post #93
danielmarkbruce wrote: post #89

Earlier quoted context omitted.

It's meaningful because the model can learn relationships between things (like, that a dog and cat are often pets and interchangdable in certain circumstances) and then produce output that it has never seen. Read the word2vec paper as an example. And, an LLM can output whatever you want. Change the last linear layer from (embedding dimension -> vocab size) to (embedding dimension -> sentiment categories) for example,…

This isn't magic. Precisely. Calling it manipulating high dimension concepts muddles the waters. It's not. It's merely using a whole lot of numbers to manipulate word tokens. And it's quite limited at that despite the impressive ability to store up language corner cases and put them back out. "an LLM can output whatever you want" -> this right here is approaching magical thinking. I am really clear headed about it's…

It is literally manipulating high dimensional vectors. GPT-4 embedding dimension is 12288(I think) and Llama-3 is 4096. Thousands of said high dimensional vectors come into the model and operations like +,-,x,/,exp,log, gelu etc etc are done on them and combinations of them. And during training there are literally geometric relationships created between the vectors based on concepts humans use. This isn't some pie in the sky assertion using hand wavy words, those are concrete statements that don't muddy any water.

It might be magical to you, it isn't magical to anyone actually working with these things that you can trivially change the last layer and have the outputs represent whatever you want. Get some samples, write a loss function, go to work. Suggesting they can only output words/tokens only displays a complete misunderstanding of how they work under the hood.

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#99

Post by namaria »

danielmarkbruce wrote: post #97
namaria wrote: post #93

Earlier quoted context omitted.

This isn't magic. Precisely. Calling it manipulating high dimension concepts muddles the waters. It's not. It's merely using a whole lot of numbers to manipulate word tokens. And it's quite limited at that despite the impressive ability to store up language corner cases and put them back out. "an LLM can output whatever you want" -> this right here is approaching magical thinking. I am really clear headed about it's…

It is literally manipulating high dimensional vectors. GPT-4 embedding dimension is 12288(I think) and Llama-3 is 4096. Thousands of said high dimensional vectors come into the model and operations like +,-,x,/,exp,log, gelu etc etc are done on them and combinations of them. And during training there are literally geometric relationships created between the vectors based on concepts humans use. This isn't some pie in…

Why you keep saying it's magical to me? You're the one describing the underlying maths as if it makes it special. To the model these 'high dimensional embeddings' are just long lists of numbers.

I've said it before and I'll repeat. Using terabytes of number lists to achieve a decent NLP isn't all that impressive.

Re: What Is ChatGPT Doing and Why Does It Work? (2023)

#100

Post by danielmarkbruce »

namaria wrote: post #99
danielmarkbruce wrote: post #97

Earlier quoted context omitted.

It is literally manipulating high dimensional vectors. GPT-4 embedding dimension is 12288(I think) and Llama-3 is 4096. Thousands of said high dimensional vectors come into the model and operations like +,-,x,/,exp,log, gelu etc etc are done on them and combinations of them. And during training there are literally geometric relationships created between the vectors based on concepts humans use. This isn't some pie in…

Why you keep saying it's magical to me? You're the one describing the underlying maths as if it makes it special. To the model these 'high dimensional embeddings' are just long lists of numbers. I've said it before and I'll repeat. Using terabytes of number lists to achieve a decent NLP isn't all that impressive.

Because you don't understand it, and it's understandable. Get a version of GPT-2 or something from huggingface. Go through layer by layer.

Anyone with brain is impressed by the state of the art LLMs. What's next? The latest GPUs aren't impressive because "shoving a bunch of electrons all around the place to do elementary arithmetic isn't impressive"? The difficulty is getting a bajillion little things to come together in a way that is useful. Pointing out that some complex thing is not impressive because it's "just lots of little simple things" is dumb as hell.

Post reply on HN