Making deep learning go brrrr from first principles (2022)
1–10 of 72 posts
Re: Making deep learning go brrrr from first principles (2022)
#2> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS
wild
Re: Making deep learning go brrrr from first principles (2022)
#3>For example, getting good performance on a dataset with deep learning also involves a lot of guesswork. But, if your training loss is way lower than your test loss, you're in the "overfitting" regime, and you're wasting your time if you try to increase the capacity of your model.
Re: Making deep learning go brrrr from first principles (2022)
#4> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS wild
Single core vs multi core accounts for much of this
Re: Making deep learning go brrrr from first principles (2022)
#5>For example, getting good performance on a dataset with deep learning also involves a lot of guesswork. But, if your training loss is way lower than your test loss, you're in the "overfitting" regime, and you're wasting your time if you try to increase the capacity of your model. https://arxiv.org/abs/1912.02292
Generally, posting a link-only reply without further elaboration comes across as a bit rude. Are you providing support for the above point? Refuting it? You felt compelled to comment, a few words to indicate what you’re actually trying to say would go a long way.
Re: Making deep learning go brrrr from first principles (2022)
#6>For example, getting good performance on a dataset with deep learning also involves a lot of guesswork. But, if your training loss is way lower than your test loss, you're in the "overfitting" regime, and you're wasting your time if you try to increase the capacity of your model. https://arxiv.org/abs/1912.02292
Generally, posting a link-only reply without further elaboration comes across as a bit rude. Are you providing support for the above point? Refuting it? You felt compelled to comment, a few words to indicate what you’re actually trying to say would go a long way.
>We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better.
Re: Making deep learning go brrrr from first principles (2022)
#7Right now, all I know how to do is pull models from Hugging Face, but someday I want to build my own small LLM from scratch
Re: Making deep learning go brrrr from first principles (2022)
#8> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS wild
This statement makes zero sense
Re: Making deep learning go brrrr from first principles (2022)
#9> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS wild
Single core vs multi core accounts for much of this
Not really. GPU many cores, at least for fp32, gives you 2 to 4 order of magnitudes compared to high speed CPU.
The rest will be from "python float" (e.g. not from numpy) to C, which gives you already 2 to 3 order of magnitude difference, and then another 2 to 3 from plan C to optimized SIMD.
See e.g. https://github.com/Avafly/optimize-gemm for how you can get 2 to 3 order of magnitude just from C.
Re: Making deep learning go brrrr from first principles (2022)
#10> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS wild
Why are we comparing a programing language and a GPU. This is a category error. Programing languages do not do any operations. They perform no FLOPs, they are the thing the FLOPs are performing.
"The I7-4770K and preform 20k more Flops than C++" is an equally sensible statement (i.e. not)