31–40 of 104 posts
[stub for offtopicness]
Minor request: can we have 'neural network' or something in the title? This is related to the machine learning 'transformer' architecture, rather than the bundle of coils that couples two circuits electromagnetically.
It's some kind of abstract machine, like a turing machine or the machine that parses regexes, isn't it?
Everything is usually a little fuzzy in a neural network. There's rarely anything like an if/else statement, although (as in the transformer example) you have some cases of "masking" values with 0 or -∞. The output is almost always fuzzy as well, being a collection of scores or probabilities. For example, a model that distinguishes cat pictures and dog pictures might emit a result like "dog:0.95 cat:0.05", and we say that it predicted a cat because the dog score is higher than the cat score.
In fact, the core of the transformer, the attention mechanism, is based on a kind of "soft lookup" operation. In a non-fuzzy system, you might want to do something like loop through each token in the sequence, check if that token is relevant to the current token, and take some action if it's relevant. But in a transformer, relevance is not a binary decision. Instead, the attention mechanism computes a continuous relevance score between each pair of tokens in the sequence, and uses those scores to take further action.
But some things are not easily generalized directly from a system based on of binary decisions. For example, those relevance scores are used as weights to compute a weighted average over tokens in the vocabulary, and thereby obtain an "average token" for the current position in the sequence. I don't think there's an easy way to interpret this as an extension of some process based on branching logic.
> maybe even feel inspired to make your own model by hand as well! Other then a learning exercise to satisfy your curiosity what are you doing with this? I'm starting to get the feeling that anything complex with ml models is unreasonable for a at home blog reader?
> maybe even feel inspired to make your own model by hand as well! Other then a learning exercise to satisfy your curiosity what are you doing with this? I'm starting to get the feeling that anything complex with ml models is unreasonable for a at home blog reader?
"I've been wanting to understand transformers and attention better for awhile now—I'd read The Illustrated Transformer, but still didn't feel like I had an intuitive understanding of what the various pieces of attention were doing. What's the difference between q and k? And don't even get me started on v!"
Earlier quoted context omitted.
I misinterpreted that as "I have no training; I am a beginner" rather than "I am not going to train this thing" :)
That's how I interpreted as well. "Electrical transformer without any formal training in electric engineering" is basically how the title read to me. Followed by a lot of confusion when the article was not that. Good on him, it's just... an odd way to phrase it :)
I've been kicking around a similar idea for awhile. Why can't we have an intuitive interface to the weights of a model, that a domain expert can tweak by hand to accelerate training? For example, in a vision model, they can increase the "orangeness" collection of weights when detecting traffic cones. That way, instead of requiring thousands/millions more examples to calibrate "orangeness" right, it's accelerated by a…
I've been kicking around a similar idea for awhile. Why can't we have an intuitive interface to the weights of a model, that a domain expert can tweak by hand to accelerate training? For example, in a vision model, they can increase the "orangeness" collection of weights when detecting traffic cones. That way, instead of requiring thousands/millions more examples to calibrate "orangeness" right, it's accelerated by a…
I've been kicking around a similar idea for awhile. Why can't we have an intuitive interface to the weights of a model, that a domain expert can tweak by hand to accelerate training? For example, in a vision model, they can increase the "orangeness" collection of weights when detecting traffic cones. That way, instead of requiring thousands/millions more examples to calibrate "orangeness" right, it's accelerated by a…