One thing people seems not to acknowledge, and this post made it super clear is that NVIDIA kept their lead extremely well in a few years of very high growth. The TFLOPs, the bandwidth, the interconnect mentioned in this post continues to grow at exponential rate with no sign of stopping yet. This is a 30-year-old incumbent reminding you. The willingness to compete from NVIDIA is just simply remarkable.
Making deep learning go brrrr from first principles (2022)
51–60 of 72 posts
Re: Making deep learning go brrrr from first principles (2022)
#52Deep learning is just glorified linear algebra. Master the progression: Feed-forward CNN RNN LSTM Attention. You don't even need a GPU to understand the climax; Karpathy’s llama2.c implements a full transformer inference engine in just ~300 lines of C using SIMD pragmas for CPU execution.
I wish more people pursued that approach to teaching neural networks. First teach what the network does and why, writing it as a loopy, inference-only Python function. Explain training only in an abstract way, E.G. with the "take a random weight, twist it a little and see if the loss improves" algorithm. This lets you focus on the architecture and on why it is what it is. Then, teach the intuitions behind derivatives…
I just assume that people who are going to do useful things in ML have basic foundation in math and science. If you don’t know what a derivative is what are we doing talking about multi-variable optimization.
And it’s not about gate-keeping it’s really about being able to reason about these concepts. What this looks like in programming is people memorizing a million clean code rules and not being able to write binary search.
Re: Making deep learning go brrrr from first principles (2022)
#53Earlier quoted context omitted.
A100 FP32 throughput “at its limit”: 19.5 TFLOP/s. AMD EPYC 9965 FP32 throughput “at its limit”: 41.2 TFLOP/s (192 cores x 64 FP32 FLOP/cycle/core x 3.35GHz).
That's also a CPU that came out four years later than the A100. The contemporaneous B200 is not optimized for FP32 and does 74.45 TFLOP/s. For FP16 it's at ~2 PFLOP/s.
The point is that modern CPUs are not as slow as most DL people think. Roughly 10x slower but with a lot more memory.
Re: Making deep learning go brrrr from first principles (2022)
#54Earlier quoted context omitted.
A100 FP32 throughput “at its limit”: 19.5 TFLOP/s. AMD EPYC 9965 FP32 throughput “at its limit”: 41.2 TFLOP/s (192 cores x 64 FP32 FLOP/cycle/core x 3.35GHz).
EPYC 9965: 614GBps of 12-channel DDR5-6400 A100: 1935GBps of HBM2e Most of those FLOPS are constrained by memory bandwidth.
> Most of those FLOPS are constrained by memory bandwidth
I believe inference with large enough batch size is almost always compute bound, simply due to algorithmic complexity.
Each step of tiled matric multiplication with square tiles of size N^2 takes O(N^2) memory loads and O(N^3) compute operations. With N = 32 or 64, you will likely saturate compute even on iGPUs with DDR4 or DDR5 memory pretending to be VRAM.
Re: Making deep learning go brrrr from first principles (2022)
#55Earlier quoted context omitted.
I wish more people pursued that approach to teaching neural networks. First teach what the network does and why, writing it as a loopy, inference-only Python function. Explain training only in an abstract way, E.G. with the "take a random weight, twist it a little and see if the loss improves" algorithm. This lets you focus on the architecture and on why it is what it is. Then, teach the intuitions behind derivatives…
I just assume that people who are going to do useful things in ML have basic foundation in math and science. If you don’t know what a derivative is what are we doing talking about multi-variable optimization. And it’s not about gate-keeping it’s really about being able to reason about these concepts. What this looks like in programming is people memorizing a million clean code rules and not being able to write binary…
There's a wide gulf between knowing what a derivative is and proficiently working out the derivatives of arbitrary functions. The extent of understanding required for most applied ML is "rate of change".
Re: Making deep learning go brrrr from first principles (2022)
#56Deep learning is just glorified linear algebra. Master the progression: Feed-forward CNN RNN LSTM Attention. You don't even need a GPU to understand the climax; Karpathy’s llama2.c implements a full transformer inference engine in just ~300 lines of C using SIMD pragmas for CPU execution.
So you created a new account to blatantly plagiarize another comment from this same page? What's even going on here?
Re: Making deep learning go brrrr from first principles (2022)
#57Earlier quoted context omitted.
>We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better.
Does this mean that if your model is "overfitting", the solution is to train for even more epochs?
Maybe. Just means that the conventional wisdom was wrong and substantially over training can be a good thing. No one I knew at the time suspected that, including the people who wrote the paper.
Re: Making deep learning go brrrr from first principles (2022)
#58Earlier quoted context omitted.
I just assume that people who are going to do useful things in ML have basic foundation in math and science. If you don’t know what a derivative is what are we doing talking about multi-variable optimization. And it’s not about gate-keeping it’s really about being able to reason about these concepts. What this looks like in programming is people memorizing a million clean code rules and not being able to write binary…
There's a wide gulf between knowing what a derivative is and proficiently working out the derivatives of arbitrary functions. The extent of understanding required for most applied ML is "rate of change".
Is it that wide though? For example, how do you explain why you cannot autograd through sampling (and thus you use either a reparameterization trick, or gumbel). Sure, instead of relying on differentiability, you can intuitively explain it "the output changes only when you literally reach the next threshold, so all the way in between you don't really get a good direction", but how far are you going to take this?
I agree with your general point, that we don't need insane levels of math, but I would say a college level of calculus, linalg and probability is baseline.
A basic benchmark off the top of my head:
Being able to pick up, without stumbling on the fundamentals
- what LoRA is doing
- how a RBF-kernel SVM works
- why KL and reverse-KL are different
- why using mean squared error is equivalent to MLE on a gaussian
Not saying the four above pieces are all necessary, but that you should be able to learn them on demand without needing to revisit what a basis vector is.
"Working out derivatives of arbitrary functions" is school level.
Re: Making deep learning go brrrr from first principles (2022)
#59Earlier quoted context omitted.
Not really. GPU many cores, at least for fp32, gives you 2 to 4 order of magnitudes compared to high speed CPU. The rest will be from "python float" (e.g. not from numpy) to C, which gives you already 2 to 3 order of magnitude difference, and then another 2 to 3 from plan C to optimized SIMD. See e.g. https://github.com/Avafly/optimize-gemm for how you can get 2 to 3 order of magnitude just from C.
Theoretical FP32 performance of AMD EPYC 9965 is double that of A100: 41.2 TFLOP/s vs 19.5 TFLOP/s
Isn't that because the A100 is optimizing for memory bandwidth per TF?
Re: Making deep learning go brrrr from first principles (2022)
#60Earlier quoted context omitted.
There's a wide gulf between knowing what a derivative is and proficiently working out the derivatives of arbitrary functions. The extent of understanding required for most applied ML is "rate of change".
Is it that wide though? For example, how do you explain why you cannot autograd through sampling (and thus you use either a reparameterization trick, or gumbel). Sure, instead of relying on differentiability, you can intuitively explain it "the output changes only when you literally reach the next threshold, so all the way in between you don't really get a good direction", but how far are you going to take this? I ag…
Rate of change -> it is flat -> that is not a useful signal. I don't see the issue?
We aren't talking about doing cutting edge research, just educating people on the basics of how ML does what it does. I agree that the things you list should follow at some point in the sequence for any rigorous education. But it's a question of at what point those things should come up and what the corresponding depth of education is.
For the initial introduction I think everything you listed is entirely out of scope. You don't need any of that to get a basic MLP working using a for loop and naive gradient descent.