Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What are transformer models and how do they work?

txt.cohere.ai

Post reply on HN
Print view View original

31–40 of 114 posts

Re: What are transformer models and how do they work?

#31

Post by microtonal »

chaxor wrote: post #18
jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

I know we don't have access to the details at OpenAI - but it does seem like there have been significant changes to the BPE token size over time. It seems there is a push towards much larger tokens than the previous ~3 char tokens (at least by behavior)

BPE is not set to a certain length, but a target vocabulary size. It starts with bytes (or characters) as the basic unit in which everything is split up and merges units iteratively (choosing the most frequent pairing) until the vocab size is reached. Even 'old' BPE models contain plenty of full tokens. E.g. RoBERTa:

https://huggingface.co/roberta-base/raw/main/merges.txt

(You have to scroll down a bit to get to the larger merges and image the lines without the spaces, which is what a string would look like after a merge.)

Also see GPT-2:

https://huggingface.co/gpt2/raw/main/merges.txt

I recently did some statistics. Average number of pieces per token (sampled on fairly large data, these are all models that use BBPE):

RoBERTa base (English): 1.08

RobBERT (Dutch): 1.21

roberta-base-ca-v2 (Catalan): 1.12

ukr-models/xlm-roberta-base-uk (Ukrainian): 1.68

In all these cases, the median token length in pieces was 1.

(Note: I am not debating that newer OpenAI models don't use a larger vocab. I just want to show that older BBPE models didn't use 3 char pieces. They were 1 piece per token for most tokens.)

Re: What are transformer models and how do they work?

#32

Post by Buttons840 »

antimora wrote: post #11
jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

I am trying to learn more in depth. Could you suggest some good resource for learning transformers?

I've been reading this paper with pseudocode for various transformers and finding it helfpul: https://arxiv.org/abs/2207.09238

"This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (not results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models."

Re: What are transformer models and how do they work?

#34

Post by seydor »

I don't think anyone knows yet why transformers work. "Attention is moving vectors in embedding space" does not make sense. At the moment we know how they multiply vectors and then pass through networks etc, but let's not pretend that we understand how they "work".

Re: What are transformer models and how do they work?

#36

Post by pmoriarty »

p-e-w wrote: post #24

The more I learn about the technical details of how ML systems are implemented, the more I feel that those details obscure , rather than illuminate, what is actually going on. It's as if we were trying to understand human ethics by looking at neurotransmitters or synapses in the brain. These structures seem way too low-level to actually explain the interesting stuff. What I hear is "something something transformer au…

This is called emergent behavior, and we don't know how it happens with organic brains, minds, and neurons either.

It's actually pretty amazing that it's happening at all with computers, since neural nets are such simple, high level abstractions compared to how the brain works.

It's possible that all the tremendous complexity of organic systems isn't actually necessary for intelligence or consciousnes, which is similarly surprising.

Re: What are transformer models and how do they work?

#37

Post by detrites »

antimora wrote: post #11
jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

I am trying to learn more in depth. Could you suggest some good resource for learning transformers?

This hour-long MIT lecture is very good, it builds from the ground up until transformers. MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention: https://youtube.com/watch?v=ySEx_Bqxvvo

Re: What are transformer models and how do they work?

#38

Post by oergiR »

jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

I agree except for (6). A language model assigns probabilities to sequences. The model needs normalised distributions, eg using a softmax, so that’s the right way of thinking about it.

Re: What are transformer models and how do they work?

#39

Post by fruit2020 »

pmoriarty wrote: post #36
p-e-w wrote: post #24

The more I learn about the technical details of how ML systems are implemented, the more I feel that those details obscure , rather than illuminate, what is actually going on. It's as if we were trying to understand human ethics by looking at neurotransmitters or synapses in the brain. These structures seem way too low-level to actually explain the interesting stuff. What I hear is "something something transformer au…

This is called emergent behavior, and we don't know how it happens with organic brains, minds, and neurons either. It's actually pretty amazing that it's happening at all with computers, since neural nets are such simple, high level abstractions compared to how the brain works. It's possible that all the tremendous complexity of organic systems isn't actually necessary for intelligence or consciousnes, which is simil…

Do you mean that the gpt creators cannot backtrack an answer to understand how the model came up with it? If it’s such a black box how do they evolve it? Trial and error?

Re: What are transformer models and how do they work?

#40

Post by PeterisP »

riffraff wrote: post #28
1024core wrote: post #10

> Transformers are a new development in machine learning that have been making a lot of noise lately. The paper on transformers was published 6 years ago. 6 years in ML is an eternity nowadays.

Was there any "big idea" after that? It seems most of the user-visible innovation has been "let's use transformers on more data". Perhaps capsule networks? But those are years old too.

No, not really, there was a lot of engineering work and bunch of not-so-big ideas (e.g. InstructGPT reinforcement learning after the model's training), but you can go from the transformers paper to current state of art without needing a "big idea".

And I think this is the major "big idea", accepting the bitter lesson (http://incompleteideas.net/IncIdeas/BitterLesson.html) that major user-visible progress and new emerging capabilities doesn't necessarily require any big ideas but simply scaling to more compute.

Post reply on HN