11–20 of 72 posts
Right now, all I know how to do is pull models from Hugging Face, but someday I want to build my own small LLM from scratch
> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS wild
Right now, all I know how to do is pull models from Hugging Face, but someday I want to build my own small LLM from scratch
Right now, all I know how to do is pull models from Hugging Face, but someday I want to build my own small LLM from scratch
Earlier quoted context omitted.
Generally, posting a link-only reply without further elaboration comes across as a bit rude. Are you providing support for the above point? Refuting it? You felt compelled to comment, a few words to indicate what you’re actually trying to say would go a long way.
>We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better.
Earlier quoted context omitted.
>We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better.
Right, isn't double descent one of the reasons why modern Extremely Large Language Models work at all? I think I heard somewhere that basically all today's "smart" (reasoning, solving math problems, etc) LLMs are trained in the "double descent" territory (whatever this means, I'm not entirely sure).
> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS wild
yes of course this is apples to oranges but that's kind of the point
it shows the vast span between specialized hardware throughput IFF you can use an A100 at its limit vs overhead of one of the most popular programming languages in use today that eventually does the "same thing" on a CPU
the interesting thing is why that is so
CPU vs GPU (latency vs throughput), boxing vs dense representation, interpreter overhead, scalar execution, layers upon layers, …
> in the time that Python can perform a single FLOP, an A100 could have chewed through 9.75 million FLOPS wild
Why are we comparing a programing language and a GPU. This is a category error. Programing languages do not do any operations. They perform no FLOPs, they are the thing the FLOPs are performing. "The I7-4770K and preform 20k more Flops than C++" is an equally sensible statement (i.e. not)
but I find it illuminating to compare what a certain hardware can do in principle (what is possible) vs what I can "reach" as programmer within a certain system/setup
in this case NVIDIA A100 vs "Python" that does not reach a A100 (without the help of CUDA and PyTorch)
another analogy:
I find it useful to be able to compare what the fastest known way is to move a container from A to B using a certain vehicle (e.g. truck) and how that compares to how fast a person that can not drive that truck can do it + variants of it (on foot, using a cargo bike, using a boat via waterway, …)
I'm also interested in how much energy is needed, how much the hw costs and so on
Often there are many ways to do things, comparing is a great starting point for learning more
Earlier quoted context omitted.
Why are we comparing a programing language and a GPU. This is a category error. Programing languages do not do any operations. They perform no FLOPs, they are the thing the FLOPs are performing. "The I7-4770K and preform 20k more Flops than C++" is an equally sensible statement (i.e. not)
the sentence is ambiguous because "Python" can mean python + a certain library and even a different Python implementation but I find it illuminating to compare what a certain hardware can do in principle (what is possible) vs what I can "reach" as programmer within a certain system/setup in this case NVIDIA A100 vs "Python" that does not reach a A100 (without the help of CUDA and PyTorch) another analogy: I find it u…
that said: Python can get to more FLOPs by changing the representation: https://docs.python.org/3/library/array.html