Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT Explained: A normie's guide to how it works

jonstokes.com

Post reply on HN
Print view View original

141–144 of 144 posts

Re: ChatGPT Explained: A normie's guide to how it works

#141

Post by spion »

cwkoss wrote: post #131
spion wrote: post #5

I like the token window bit. I don't really like the probability bit, because it kinda alludes that OpenAI just built a huge probability map of all N-grams (N=8000) and called it a day. Which incidentally would also imply that a lot of N-grams just don't exist in the training data, causing the model to completely halt when someone says something unexpected. But that's not the case - instead we convert words into a lo…

How many dimensions does the vector space have? Seems like there are an infinite number of ways you could convert word->vector, how do you choose which one is 'right'? What should I google to understand how a word is encoded as a vector and then vector turned back into word(s)?

That's the beauty of it - you don't have to choose, the network chooses for you. If you set up your layers such that the first input layer has one input for each possible token (lets say total of 80K weights), and the next layer connected to that layer only 1024 values, you will get "word embeddings" as a consequence of training the network (i.e. each node in the 80K layer will have 1024 values pointing to individual nodes of the next layer)

The simplest way to get word embeddings (without necessarily building a complex GPT like model) is word2vec: https://towardsdatascience.com/creating-word-embeddings-codi... - the principle is similar but the network is smaller.

Re: ChatGPT Explained: A normie's guide to how it works

#142

Post by oblib »

Silverback_VII wrote: post #40
kat_rebelo wrote: post #15

Earlier quoted context omitted.

i'm not a red state / far right / pro-trump in any sense of the word. however, i don't think it is very unreasonable to extrapolate bit and see the potential for societal harm. over the past three years the entire world was impacted by a dire health crisis where misinformation played a large role in distorting public perception. this has direct impacts on public health (people not wearing masks, refusing vaccines) an…

>misinformation played a large role in distorting public perception. this has direct impacts on public health (people not wearing masks, refusing vaccines) I increasingly hear things that suggest that those who wore masks and got vaccinated were the ones who were actually misinformed. Of course, you won't hear any of that on CNN

> I increasingly hear things that suggest those who wore masks and got vaccinated were the ones who were actually misinformed

The only folks who were misinformed are those who never took the time to learn about masks and vaccinations. But to be fair that is a huge number of folks here in the U.S.

I've yet hear anyone talk about someone they infected who was killed by it though. With over 1.1 Million dead from it here in the U.S. that's astonishing. So is the fact that it is still killing around 2000+ a week here.

Re: ChatGPT Explained: A normie's guide to how it works

#144

Post by jkbyc »

mirekrusin wrote: post #111
phire wrote: post #68

Earlier quoted context omitted.

Worth pointing out that it's not 100% accurate at arithmetic, that becomes quite clear when you ask it to calculate large numbers. It's result is usually close, but not perfect. So it hasn't learned the exact rules for arithmetic, it's learned rules for approximating arithmetic to a decent level of accuracy. Similar to how humans can know the approximate result of an equation before doing the actual math (though GPT…

They need to add support for co-brains - functions that are available for weightware brain to use. Ie if could and knew how to use wolfram alpha - it’d boost performance dramatically.

That's called Toolformer (Facebook AI Research): https://arxiv.org/abs/2302.04761 Toolformer: Language Models Can Teach Themselves to Use Tools
Post reply on HN