Does the human brain use transformers?
Understand how transformers work by demystifying the math behind them
11–20 of 139 posts
Re: Understand how transformers work by demystifying the math behind them
#12Earlier quoted context omitted.
This is basically this - it can learn ignore some paths, and amplify something more important, then you can just cut this paths without sensible loss of quality. The problem is that you are not going to win anything from this - non-matrix multiplication would be slower or the same.
The issue is that you are thinking of this in terms of information compression, which is what LLMs are. Im more concerned with an LLM having the ability to be trained to the point where a subset of the graph represents all the nand gates necessary for a cpu and ram, so when you ask it questions it can actually run code to compute them accurately instead of offering a statistical best guess, i.e decompression after lo…
Just give it a computer? Even a virtual machine. It can output assembly code or high level code that gets compiled.
Re: Understand how transformers work by demystifying the math behind them
#13There I was all excited to show off some of my electrical chops on HN. Not today.
Does mystified math lie beyond behind how the ratio of input and output voltages is equal to the ratio of the primary and secondary windings? Can it be derived from Maxwell's equations?
Off to a search...
Re: Understand how transformers work by demystifying the math behind them
#14Does the human brain use transformers?
No, but they can both implement a language virtual machine which appears to be able to produce intelligent behaviour with unknown bounds.
Re: Understand how transformers work by demystifying the math behind them
#15Does the human brain use transformers?
Yes, through ie. services like openai's chatgpt.
Re: Understand how transformers work by demystifying the math behind them
#16Do LLMs use neural nets? If so, what makes up the "neuron"? i.e. Is there a code structure that underlies the neuron, or is it "just" fancy math?
Re: Understand how transformers work by demystifying the math behind them
#17There I was all excited to show off some of my electrical chops on HN. Not today.
Does mystified math lie beyond behind how the ratio of input and output voltages is equal to the ratio of the primary and secondary windings? Can it be derived from Maxwell's equations? Off to a search...
I bet that an LLM (which uses transformers) can explain those aspects of a transformer to you.
Re: Understand how transformers work by demystifying the math behind them
#18For a dryer, more formal and succinct approach, see "The Transformer Model in Equations" [0], by
John Thickstun. The whole thing fits in a single page, using standard mathematical notation.
Re: Understand how transformers work by demystifying the math behind them
#19Transformer tutorials might be the new monad tutorial. A hard concept to get, but one you need to struggle with (and practice some examples) to understand. So a bit like much of computer science :-).
Waiting for a blogpost titled "You could have invented transformers".
Re: Understand how transformers work by demystifying the math behind them
#20Do LLMs use neural nets? If so, what makes up the "neuron"? i.e. Is there a code structure that underlies the neuron, or is it "just" fancy math?
Yes to both, the "neuron" would basically be a weighted parameter.
A parameter is an expression, it's a mathematical representation of a token and it's probabilistic weighting (theyre translated from input or to output token lists entering and exiting the model).
Usually tokens are pre-set small groups of character combinations like "if " or "cha" that make up a word/sentence.
The recorded path your value takes down the chain of probabilities would be the "neural pathway" within the wider "neural network".
Someone please correct me if I'm wrong or my terminology is wrong.