Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What are transformer models and how do they work?

txt.cohere.ai

Post reply on HN
Print view View original

21–30 of 114 posts

Re: What are transformer models and how do they work?

#21

Post by hansvm »

jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

> 6

Selecting the likeliest token is only one of many sampling options, and it's extremely poor for most tasks, moreso when you consider the relationships between multiple executions of the model. _Some_ (not necessarily softmax) probability renormalization trained into the model is issential for a lot of techniques.

Re: What are transformer models and how do they work?

#23

Post by quantisan »

posharma wrote: post #3

I found the blogs written by Jay Alammar to be much more informative and complete. It appears that companies are rehashing and compressing the same content to advertise their products.

https://jalammar.github.io/visualizing-neural-machine-transl... and https://jalammar.github.io/illustrated-transformer/ for anyone looking

Re: What are transformer models and how do they work?

#24

Post by p-e-w »

The more I learn about the technical details of how ML systems are implemented, the more I feel that those details obscure, rather than illuminate, what is actually going on.

It's as if we were trying to understand human ethics by looking at neurotransmitters or synapses in the brain. These structures seem way too low-level to actually explain the interesting stuff.

What I hear is "something something transformer autoencoder attention [...] MAGIC [...] a machine that speaks like a human".

Where is the connection between computational details and the model's high-level behavior? Do we even know? Is there a "psychology of ML models" that develops useful concepts that deal with what a model does, rather than how it functions at the plumbing layer?

Re: What are transformer models and how do they work?

#25

Post by asdfman123 »

p-e-w wrote: post #24

The more I learn about the technical details of how ML systems are implemented, the more I feel that those details obscure , rather than illuminate, what is actually going on. It's as if we were trying to understand human ethics by looking at neurotransmitters or synapses in the brain. These structures seem way too low-level to actually explain the interesting stuff. What I hear is "something something transformer au…

I agree.

Maybe it's because the human mind is good at breaking things into neat modules that fit together hierarchically. We can figure them out piecemeal and eventually grasp the whole system. But messy organic systems are not like that, and we just don't have the hardware to perceive everything at once.

Or maybe it's because we have trouble acknowledging that intelligence and consciousness isn't limited to animals, and the human brain doesn't have to epitomize it.

Re: What are transformer models and how do they work?

#26

Post by SanderNL »

p-e-w wrote: post #24

The more I learn about the technical details of how ML systems are implemented, the more I feel that those details obscure , rather than illuminate, what is actually going on. It's as if we were trying to understand human ethics by looking at neurotransmitters or synapses in the brain. These structures seem way too low-level to actually explain the interesting stuff. What I hear is "something something transformer au…

We just don’t know. We can’t even agree on if there is any significant high-level behavior.

“It’s alive!” and “stochastic parrot” are still both quite popular in my experience.

Re: What are transformer models and how do they work?

#27

Post by VMG »

jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

I have to agree. The article summary says

> Transformer block: Guesses the next word. It is formed by an attention block and a feedforward block.

But the diagram shows transformer blocks chained in sequence. So the next transformer block in the sequence would only receive a single word as the input? Does not make sense.

Re: What are transformer models and how do they work?

#28

Post by riffraff »

1024core wrote: post #10

> Transformers are a new development in machine learning that have been making a lot of noise lately. The paper on transformers was published 6 years ago. 6 years in ML is an eternity nowadays.

Was there any "big idea" after that?

It seems most of the user-visible innovation has been "let's use transformers on more data".

Perhaps capsule networks? But those are years old too.

Re: What are transformer models and how do they work?

#30

Post by riffraff »

jaidhyani wrote: post #9
dilawar wrote: post #8

Thanks. ML noob here. I liked the insight that attention adds context, by modifying the distance in an embedding.

The way the article presents this is misleading. The attention mechanism builds a new vector as a linear combination of other vectors, but after the first layer these have also all been altered by passing through a transformer layer so it makes less sense to talk about "other tokens" in most cases (it becomes increasingly inaccurate the deeper into the model you go). It's also not really moving closer so much as addi…

Isn't "adding" the same as "moving closer" ?

E.g. the vector for "bank" is mid-way between the geographical and financial meaning, "bank + money" is closer while "bank + river" if further away.

Post reply on HN