Making deep learning go brrrr from first principles (2022) • Hacker News Board

Re: Making deep learning go brrrr from first principles (2022)

#21

Post by p1esk » Sat, May 23, 2026, 2:33 PM UTC

tosh wrote: post #18

tosh wrote: post #2

> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS wild

re comments: yes of course this is apples to oranges but that's kind of the point it shows the vast span between specialized hardware throughput IFF you can use an A100 at its limit vs overhead of one of the most popular programming languages in use today that eventually does the "same thing" on a CPU the interesting thing is why that is so CPU vs GPU (latency vs throughput), boxing vs dense representation, interpret…

A100 FP32 throughput “at its limit”: 19.5 TFLOP/s.

AMD EPYC 9965 FP32 throughput “at its limit”: 41.2 TFLOP/s (192 cores x 64 FP32 FLOP/cycle/core x 3.35GHz).

Re: Making deep learning go brrrr from first principles (2022)

#22

Post by gchamonlive » Sat, May 23, 2026, 2:36 PM UTC

patmorgan23 wrote: post #10

tosh wrote: post #2

> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS wild

Why are we comparing a programing language and a GPU. This is a category error. Programing languages do not do any operations. They perform no FLOPs, they are the thing the FLOPs are performing. "The I7-4770K and preform 20k more Flops than C++" is an equally sensible statement (i.e. not)

> Why are we comparing a programing language and a GPU.

You are taking the statement too literally and forgetting it's a figure of speech, specifically metonymy.

When the author says it's millions of flops faster in a gpu than in an interpreteted programming language, it's not comparing them directly, but algorithms that run in them, so the substitution is the algorithms for the tools used to implement/run them.

It makes sense if you say "running similar logic -- like multiplying vectors and matrices -- on the CPU is millions of flops slower then on the GPU". There is no category error there.

Re: Making deep learning go brrrr from first principles (2022)

#23

Post by p1esk » Sat, May 23, 2026, 2:37 PM UTC

cdavid wrote: post #9

xyzsparetimexyz wrote: post #4

Earlier quoted context omitted.

Single core vs multi core accounts for much of this

Not really. GPU many cores, at least for fp32, gives you 2 to 4 order of magnitudes compared to high speed CPU. The rest will be from "python float" (e.g. not from numpy) to C, which gives you already 2 to 3 order of magnitude difference, and then another 2 to 3 from plan C to optimized SIMD. See e.g. https://github.com/Avafly/optimize-gemm for how you can get 2 to 3 order of magnitude just from C.

Theoretical FP32 performance of AMD EPYC 9965 is double that of A100: 41.2 TFLOP/s vs 19.5 TFLOP/s

Re: Making deep learning go brrrr from first principles (2022)

#24

Post by tosh » Sat, May 23, 2026, 2:46 PM UTC

p1esk wrote: post #21

tosh wrote: post #18

Earlier quoted context omitted.

re comments: yes of course this is apples to oranges but that's kind of the point it shows the vast span between specialized hardware throughput IFF you can use an A100 at its limit vs overhead of one of the most popular programming languages in use today that eventually does the "same thing" on a CPU the interesting thing is why that is so CPU vs GPU (latency vs throughput), boxing vs dense representation, interpret…

A100 FP32 throughput “at its limit”: 19.5 TFLOP/s. AMD EPYC 9965 FP32 throughput “at its limit”: 41.2 TFLOP/s (192 cores x 64 FP32 FLOP/cycle/core x 3.35GHz).

A100: 312 TFLOP/s for FP16

but it is very impressive how far modern CPUs get as well (also in smart phones!)

Re: Making deep learning go brrrr from first principles (2022)

#25

Post by smallerize » Sat, May 23, 2026, 2:53 PM UTC

noosphr wrote: post #6

appplication wrote: post #5

Earlier quoted context omitted.

Generally, posting a link-only reply without further elaboration comes across as a bit rude. Are you providing support for the above point? Refuting it? You felt compelled to comment, a few words to indicate what you’re actually trying to say would go a long way.

>We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better.

Does this mean that if your model is "overfitting", the solution is to train for even more epochs?

Re: Making deep learning go brrrr from first principles (2022)

#26

Post by itishappy » Sat, May 23, 2026, 2:57 PM UTC

tosh wrote: post #2

> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS wild

Which, lets be honest, is probably still being orchestrated by Python somewhere.

Python is 9.75 million times faster than Python.

Re: Making deep learning go brrrr from first principles (2022)

#27

Post by giancarlostoro » Sat, May 23, 2026, 3:01 PM UTC

itishappy wrote: post #26

tosh wrote: post #2

> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS wild

Which, lets be honest, is probably still being orchestrated by Python somewhere. Python is 9.75 million times faster than Python.

I was researching if there was much benefit to using Rust or C++ over Python for AI, and turns out, the GPU doesn't care once the instructions are in because its an entirely different spec running on the GPU. The only thing you might save on is "startup" costs of getting your code into the GPU I guess? I assume that time cost is miniscule though, once its all in memory, nobody cares that you spent any time "booting it up" any more than how long Windows takes these days.

Re: Making deep learning go brrrr from first principles (2022)

#28

Post by p1esk » Sat, May 23, 2026, 3:11 PM UTC

tosh wrote: post #24

p1esk wrote: post #21

Earlier quoted context omitted.

A100 FP32 throughput “at its limit”: 19.5 TFLOP/s. AMD EPYC 9965 FP32 throughput “at its limit”: 41.2 TFLOP/s (192 cores x 64 FP32 FLOP/cycle/core x 3.35GHz).

A100: 312 TFLOP/s for FP16 but it is very impressive how far modern CPUs get as well (also in smart phones!)

Intel Xeon 6980P: 128 cores x 1024 FP16 FLOP/cycle/core x 3.2 GHz: 419 TFLOP/s

Re: Making deep learning go brrrr from first principles (2022)

#29

Post by smasher164 » Sat, May 23, 2026, 3:52 PM UTC

patmorgan23 wrote: post #10

tosh wrote: post #2

> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS wild

Why are we comparing a programing language and a GPU. This is a category error. Programing languages do not do any operations. They perform no FLOPs, they are the thing the FLOPs are performing. "The I7-4770K and preform 20k more Flops than C++" is an equally sensible statement (i.e. not)

> This is a category error.

Okay, but surely you know what they actually mean right, or are you being willfully obtuse? They are comparing CPython (the main python implementation)'s implementation that runs on the CPU with a kernel running on the GPU.

Re: Making deep learning go brrrr from first principles (2022)

#30

Post by BillStrong » Sat, May 23, 2026, 3:53 PM UTC

giancarlostoro wrote: post #27

itishappy wrote: post #26

Earlier quoted context omitted.

Which, lets be honest, is probably still being orchestrated by Python somewhere. Python is 9.75 million times faster than Python.

I was researching if there was much benefit to using Rust or C++ over Python for AI, and turns out, the GPU doesn't care once the instructions are in because its an entirely different spec running on the GPU. The only thing you might save on is "startup" costs of getting your code into the GPU I guess? I assume that time cost is miniscule though, once its all in memory, nobody cares that you spent any time "booting i…

As long as you don't keep calling out to the CPU, that is.

Tool calling, searches, cache movement if used, and even debug steps all stall the GPU waiting for the CPU.

There was a test of turning one of the under 1B Qwen3+ models into a kernel that didn't stall by the CPU as one GPU pass that saw quite a bit f perf lift over vLLM, I believe, showing this is an issue still.

Its been a month, so I don't remember more details than this.