Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making deep learning go brrrr from first principles (2022)

horace.io

Post reply on HN
Print view View original

31–40 of 72 posts

Re: Making deep learning go brrrr from first principles (2022)

#31

Post by tosh »

p1esk wrote: post #28
tosh wrote: post #24

Earlier quoted context omitted.

A100: 312 TFLOP/s for FP16 but it is very impressive how far modern CPUs get as well (also in smart phones!)

Intel Xeon 6980P: 128 cores x 1024 FP16 FLOP/cycle/core x 3.2 GHz: 419 TFLOP/s

I'm not saying "GPU more brrt than CPU"

I found the comparison interesting

on Intel Xeon 690P with 419 TFLOP/s it is still (maybe even more?) interesting to ask:

how much throughput can you reach with Python, Python with lib x, y, z, with C++ like this, with C++ like that etc etc and why?

no?

Re: Making deep learning go brrrr from first principles (2022)

#32

Post by bee_rider »

smasher164 wrote: post #29
patmorgan23 wrote: post #10

Earlier quoted context omitted.

Why are we comparing a programing language and a GPU. This is a category error. Programing languages do not do any operations. They perform no FLOPs, they are the thing the FLOPs are performing. "The I7-4770K and preform 20k more Flops than C++" is an equally sensible statement (i.e. not)

> This is a category error. Okay, but surely you know what they actually mean right, or are you being willfully obtuse? They are comparing CPython (the main python implementation)'s implementation that runs on the CPU with a kernel running on the GPU.

I’m not 100%, in context. Sorry for the big quote:

> Overhead is when your code is spending time doing anything that's not transferring tensors or computing things. For example, time spent in the Python interpreter? Overhead. Time spent in the PyTorch framework? Overhead. Time spent launching CUDA kernels (but not executing them)? Also... overhead.

> The primary reason overhead is such a pernicious problem is that modern GPUs are really fast. An A100 can perform 312 trillion floating point operations per second (312 TeraFLOPS). In comparison, Python is really slooooowwww. Benchmarking locally, Python can perform 32 million additions in one second.

> That means that in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS.

> Even worse, the Python interpreter isn't even the only source of overhead - frameworks like PyTorch also have many layers of dispatch before you get to your actual kernel. If you perform the same experiment with PyTorch, we can only get 280 thousand operations per second. Of course, tiny tensors aren't what PyTorch is built for, but... if you are using tiny tensors (such as in scientific computing), you might find PyTorch incredibly slow compared to C++.

Emphasis mine.

It’s all a bit jumbled up. I get that he was going for an informal tone and this isn’t exactly a benchmark. But I’m still not sure, based on the second emphasized part I think the “bad” measurements are coming from Python+PyTorch but with too-small workloads, and dispatching to CPU, maybe? But the first one looks like naive Python loops.

Re: Making deep learning go brrrr from first principles (2022)

#33

Post by mxwsn »

ForceBru wrote: post #16
noosphr wrote: post #6

Earlier quoted context omitted.

>We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better.

Right, isn't double descent one of the reasons why modern Extremely Large Language Models work at all? I think I heard somewhere that basically all today's "smart" (reasoning, solving math problems, etc) LLMs are trained in the "double descent" territory (whatever this means, I'm not entirely sure).

No, there are more training tokens than parameters in LLMs. They are in the classical first descent setting.

Re: Making deep learning go brrrr from first principles (2022)

#35

Post by p1esk »

tosh wrote: post #31
p1esk wrote: post #28

Earlier quoted context omitted.

Intel Xeon 6980P: 128 cores x 1024 FP16 FLOP/cycle/core x 3.2 GHz: 419 TFLOP/s

I'm not saying "GPU more brrt than CPU" I found the comparison interesting on Intel Xeon 690P with 419 TFLOP/s it is still (maybe even more?) interesting to ask: how much throughput can you reach with Python, Python with lib x, y, z, with C++ like this, with C++ like that etc etc and why? no?

No one in their right mind would use pure Python to do matrix multiplication. It’s like using a screwdriver to hammer nails into wood.

But this discussion is even more bizarre than comparing a screwdriver to a hammer, it’s like comparing a screwdriver to a nail.

Re: Making deep learning go brrrr from first principles (2022)

#38

Post by lancekey »

jdw64 wrote: post #14
kflansburg wrote: post #13

Earlier quoted context omitted.

If you aren't already aware, Karpathy has several videos that could get you there in a few hours https://www.youtube.com/@AndrejKarpathy

very thanks!

Also check out his nanochat repo. I used the repo, claude and shadeform to train my own mini model for about $300. Would have been less but I screwed up and let the cloud gpu rental run for a few hours even though the training run errored out.

Of course the model was dumber than GPT2 but still it was a great learning experience.

Re: Making deep learning go brrrr from first principles (2022)

#39

Post by vrm »

big-chungus4 wrote: post #34

How does x.cos().cos() work faster than doing two cos calls separately? Like the first cos call returns a tensor either way, the only difference is that it's not assigned to a variable. But how is it even possible know that difference in python?

It’s really not a concept you can express in idiomatic Python very easily. This comes from the actual generated assembly involving copies from global GPU memory into registers (slow, bandwidth saturates quickly) and back in between the cosines. If you can avoid the intermediate roundtrip that cuts the cost approximately in half.

Re: Making deep learning go brrrr from first principles (2022)

#40

Post by ollin »

big-chungus4 wrote: post #34

How does x.cos().cos() work faster than doing two cos calls separately? Like the first cos call returns a tensor either way, the only difference is that it's not assigned to a variable. But how is it even possible know that difference in python?

Yeah, that part should not be read literally; `x.cos().cos()` and `x1 = x.cos(); x2 = x1.cos()` both launch the same number of kernels (two in unfused/eager mode, one in fused/torch.compile, see this test notebook [1]). I think the author chained the two cos calls to symbolize the idea of combining them (without exposing the intermediate result), but chaining the two cos calls doesn't literally trigger operator fusion.

[1] https://colab.research.google.com/drive/13a4Y-ko6QLMPAhBz64c...

Post reply on HN