[flagged]
Making deep learning go brrrr from first principles (2022)
61–70 of 72 posts
Re: Making deep learning go brrrr from first principles (2022)
#62Earlier quoted context omitted.
Is it that wide though? For example, how do you explain why you cannot autograd through sampling (and thus you use either a reparameterization trick, or gumbel). Sure, instead of relying on differentiability, you can intuitively explain it "the output changes only when you literally reach the next threshold, so all the way in between you don't really get a good direction", but how far are you going to take this? I ag…
Rate of change -> it is flat -> that is not a useful signal. I don't see the issue? We aren't talking about doing cutting edge research, just educating people on the basics of how ML does what it does. I agree that the things you list should follow at some point in the sequence for any rigorous education. But it's a question of at what point those things should come up and what the corresponding depth of education is…
> For the initial introduction I think everything you listed is entirely out of scope.
Who are we giving an intro to who doesn’t have 2 years of stem education?
Re: Making deep learning go brrrr from first principles (2022)
#63I feel like there is no portable advice for performance. A torch model exported as onnx is a different model. That onnx model run using onnxruntime with cuda ep is a different model than the one run with TRT ep. And even among the same runtime, depending on the target hardware and the memory available during tuning, the model behaves differently. It is a humongous mess
That's interesting as I was considering GGUF --> ONNX conversions (via Olive), but if this creates unknown distortions in the effectiveness and stability, it might be a dead-end idea.
Re: Making deep learning go brrrr from first principles (2022)
#64Earlier quoted context omitted.
EPYC 9965: 614GBps of 12-channel DDR5-6400 A100: 1935GBps of HBM2e Most of those FLOPS are constrained by memory bandwidth.
> Most of those FLOPS are constrained by memory bandwidth I believe inference with large enough batch size is almost always compute bound, simply due to algorithmic complexity. Each step of tiled matric multiplication with square tiles of size N^2 takes O(N^2) memory loads and O(N^3) compute operations. With N = 32 or 64, you will likely saturate compute even on iGPUs with DDR4 or DDR5 memory pretending to be VRAM.
Prefill (GEMM) is compute bound, decode (GEMV) is memory bound.
Re: Making deep learning go brrrr from first principles (2022)
#65How does x.cos().cos() work faster than doing two cos calls separately? Like the first cos call returns a tensor either way, the only difference is that it's not assigned to a variable. But how is it even possible know that difference in python?
The author forgot to add "fused" here, like they did in other parts of the same section.
Non-fused:
foreach i
y[i] = cos(x[i])
foreach i
z[i] = cos(y[i])
foreach i
t = cos(x[i])
z[i] = cos(t)
Re: Making deep learning go brrrr from first principles (2022)
#66Earlier quoted context omitted.
Is it that wide though? For example, how do you explain why you cannot autograd through sampling (and thus you use either a reparameterization trick, or gumbel). Sure, instead of relying on differentiability, you can intuitively explain it "the output changes only when you literally reach the next threshold, so all the way in between you don't really get a good direction", but how far are you going to take this? I ag…
Rate of change -> it is flat -> that is not a useful signal. I don't see the issue? We aren't talking about doing cutting edge research, just educating people on the basics of how ML does what it does. I agree that the things you list should follow at some point in the sequence for any rigorous education. But it's a question of at what point those things should come up and what the corresponding depth of education is…
> You don't need any of that to get a basic MLP working using a for loop and naive gradient descent.
Well sure. Your initial statement was about "most applied ML".
> Rate of change -> it is flat -> that is not a useful signal. I don't see the issue?
It's not going to be zero if you sample in your practicum setting. You're gonna get RuntimeError: element 0 doesn't require grad and doesn't have a grad_fn. So yeah.
Re: Making deep learning go brrrr from first principles (2022)
#67Earlier quoted context omitted.
> Most of those FLOPS are constrained by memory bandwidth I believe inference with large enough batch size is almost always compute bound, simply due to algorithmic complexity. Each step of tiled matric multiplication with square tiles of size N^2 takes O(N^2) memory loads and O(N^3) compute operations. With N = 32 or 64, you will likely saturate compute even on iGPUs with DDR4 or DDR5 memory pretending to be VRAM.
Prefill (GEMM) is compute bound, decode (GEMV) is memory bound.
> decode (GEMV) is memory bound
Decode with batch size 1 is GEMV. Batching makes the decode GEMM too.
Re: Making deep learning go brrrr from first principles (2022)
#68the real lesson: GPUs win on memory bandwidth not just FLOPs. batching ops keeps VRAM fed at 2TB/s instead of tripping to RAM at 50GB/s for every operation
Re: Making deep learning go brrrr from first principles (2022)
#69Earlier quoted context omitted.
I wish more people pursued that approach to teaching neural networks. First teach what the network does and why, writing it as a loopy, inference-only Python function. Explain training only in an abstract way, E.G. with the "take a random weight, twist it a little and see if the loss improves" algorithm. This lets you focus on the architecture and on why it is what it is. Then, teach the intuitions behind derivatives…
I just assume that people who are going to do useful things in ML have basic foundation in math and science. If you don’t know what a derivative is what are we doing talking about multi-variable optimization. And it’s not about gate-keeping it’s really about being able to reason about these concepts. What this looks like in programming is people memorizing a million clean code rules and not being able to write binary…
When you learn calculus, you learn three things: the intuitions behind the concepts, the formal definitions of those concepts, and the techniques to efficiently solve problems using these concepts without a computer; things like integration by parts or by substitution.
If what you want to understand is neural networks, even at a deep level, you need a very good intuitive grasp of what derivatives are (without necessarily understanding what a limit is, if you really want to show a definition, teach the infinitesimal). You also need to understand the rules of derivation, which you can relatively easily explain if you explain derivatives. You don't need other calculus concepts (like limits, sequences or integrals). You don't need the formal definitions. You don't need to solve large derivatives on paper, and you certainly don't need to be fast at it and be able to do it in a closed-book exam setting.
Re: Making deep learning go brrrr from first principles (2022)
#70Earlier quoted context omitted.
I just assume that people who are going to do useful things in ML have basic foundation in math and science. If you don’t know what a derivative is what are we doing talking about multi-variable optimization. And it’s not about gate-keeping it’s really about being able to reason about these concepts. What this looks like in programming is people memorizing a million clean code rules and not being able to write binary…
When you learn calculus, you learn three things: the intuitions behind the concepts, the formal definitions of those concepts, and the techniques to efficiently solve problems using these concepts without a computer; things like integration by parts or by substitution. If what you want to understand is neural networks, even at a deep level, you need a very good intuitive grasp of what derivatives are (without necessa…
> You don't need the formal definitions.
To me this is simply an expression of understanding.
> You don't need to solve large derivatives on paper, and you certainly don't need to be fast at it
Agreed.