Deep learning is just glorified linear algebra. Master the progression: Feed-forward CNN RNN LSTM Attention. You don't even need a GPU to understand the climax; Karpathy’s llama2.c implements a full transformer inference engine in just ~300 lines of C using SIMD pragmas for CPU execution.
Making deep learning go brrrr from first principles (2022)
41–50 of 72 posts
Re: Making deep learning go brrrr from first principles (2022)
#42Earlier quoted context omitted.
Which, lets be honest, is probably still being orchestrated by Python somewhere. Python is 9.75 million times faster than Python.
I was researching if there was much benefit to using Rust or C++ over Python for AI, and turns out, the GPU doesn't care once the instructions are in because its an entirely different spec running on the GPU. The only thing you might save on is "startup" costs of getting your code into the GPU I guess? I assume that time cost is miniscule though, once its all in memory, nobody cares that you spent any time "booting i…
Pytorch dataloaders are often horribly inefficient, a lot of stuff there can benefit from Rust/C++
Re: Making deep learning go brrrr from first principles (2022)
#43Deep learning is just glorified linear algebra. Master the progression: Feed-forward CNN RNN LSTM Attention. You don't even need a GPU to understand the climax; Karpathy’s llama2.c implements a full transformer inference engine in just ~300 lines of C using SIMD pragmas for CPU execution.
I wish more people pursued that approach to teaching neural networks.
First teach what the network does and why, writing it as a loopy, inference-only Python function. Explain training only in an abstract way, E.G. with the "take a random weight, twist it a little and see if the loss improves" algorithm. This lets you focus on the architecture and on why it is what it is.
Then, teach the intuitions behind derivatives and gradient descent. You don't need the entirety of calculus, there's no benefit to knowing how a sequence or limit works if you ) only want to understand neural networks. With autograd, you won't be manually doing derivatives of weird functions either, so intuitive understanding is a lot more important than doing dozens of traditional calculus exercises on paper like it's the 1800s. You could probably explain the little bit of calculus you need in an hour or two, even to somebody with a 12-year-old's understanding of math and a good bit of programming knowledge.
Only when people understand the training and inference, implemented with loops and descriptive variable names, teach the tensor, explain how a modern CPU and GPU works (because many programmers still think a modern computer is just a much faster 6502), and then teach the tricks we use to make it fast.
Re: Making deep learning go brrrr from first principles (2022)
#44I feel like there is no portable advice for performance. A torch model exported as onnx is a different model.
That onnx model run using onnxruntime with cuda ep is a different model than the one run with TRT ep.
And even among the same runtime, depending on the target hardware and the memory available during tuning, the model behaves differently. It is a humongous mess
Re: Making deep learning go brrrr from first principles (2022)
#45Earlier quoted context omitted.
Which, lets be honest, is probably still being orchestrated by Python somewhere. Python is 9.75 million times faster than Python.
I was researching if there was much benefit to using Rust or C++ over Python for AI, and turns out, the GPU doesn't care once the instructions are in because its an entirely different spec running on the GPU. The only thing you might save on is "startup" costs of getting your code into the GPU I guess? I assume that time cost is miniscule though, once its all in memory, nobody cares that you spent any time "booting i…
you can port anything python is doing with a couple prompts into rust/c++, including parity validation. when the barrier to migrating is that thin, you are losing money and time even continuing to talk about it. python is miserably slow, so dont let it touch any part of your system. no snakes in the house.
Re: Making deep learning go brrrr from first principles (2022)
#46Earlier quoted context omitted.
re comments: yes of course this is apples to oranges but that's kind of the point it shows the vast span between specialized hardware throughput IFF you can use an A100 at its limit vs overhead of one of the most popular programming languages in use today that eventually does the "same thing" on a CPU the interesting thing is why that is so CPU vs GPU (latency vs throughput), boxing vs dense representation, interpret…
A100 FP32 throughput “at its limit”: 19.5 TFLOP/s. AMD EPYC 9965 FP32 throughput “at its limit”: 41.2 TFLOP/s (192 cores x 64 FP32 FLOP/cycle/core x 3.35GHz).
[deleted]
Re: Making deep learning go brrrr from first principles (2022)
#47[deleted]
Re: Making deep learning go brrrr from first principles (2022)
#48Earlier quoted context omitted.
re comments: yes of course this is apples to oranges but that's kind of the point it shows the vast span between specialized hardware throughput IFF you can use an A100 at its limit vs overhead of one of the most popular programming languages in use today that eventually does the "same thing" on a CPU the interesting thing is why that is so CPU vs GPU (latency vs throughput), boxing vs dense representation, interpret…
A100 FP32 throughput “at its limit”: 19.5 TFLOP/s. AMD EPYC 9965 FP32 throughput “at its limit”: 41.2 TFLOP/s (192 cores x 64 FP32 FLOP/cycle/core x 3.35GHz).
EPYC 9965: 614GBps of 12-channel DDR5-6400
A100: 1935GBps of HBM2e
Most of those FLOPS are constrained by memory bandwidth.
Re: Making deep learning go brrrr from first principles (2022)
#49Earlier quoted context omitted.
re comments: yes of course this is apples to oranges but that's kind of the point it shows the vast span between specialized hardware throughput IFF you can use an A100 at its limit vs overhead of one of the most popular programming languages in use today that eventually does the "same thing" on a CPU the interesting thing is why that is so CPU vs GPU (latency vs throughput), boxing vs dense representation, interpret…
A100 FP32 throughput “at its limit”: 19.5 TFLOP/s. AMD EPYC 9965 FP32 throughput “at its limit”: 41.2 TFLOP/s (192 cores x 64 FP32 FLOP/cycle/core x 3.35GHz).
That's also a CPU that came out four years later than the A100. The contemporaneous B200 is not optimized for FP32 and does 74.45 TFLOP/s. For FP16 it's at ~2 PFLOP/s.
Re: Making deep learning go brrrr from first principles (2022)
#50I'd want to see more about the failure modes. Production systems need graceful degradation more than optimal performance.