Numbers every LLM Developer should know
anyscale.com
Numbers every LLM Developer should know
1–10 of 20 posts
Re: Numbers every LLM Developer should know
#2This will not age well.
Re: Numbers every LLM Developer should know
#3This is honestly a bit gross, as it's just a marketing piece.
The original numbers every programmer should know is a profound piece of pedagogy, aimed at helping programmers be better at their craft.
This is just an excerpt from a pitch deck.
Re: Numbers every LLM Developer should know
#4What’s this “neural information retrieval system” thing about?
I’m just hacking away and presenting the LLM with some JSON data from our metrics database and making it answer user questions as a completion.
Is this embedding thing relevant for what I’m doing? Where should I start reading?
Re: Numbers every LLM Developer should know
#5I clicked because I thought they were defining LLM developer as "someone training LLMs", but instead they define it as "someone integrating LLMs into their application".
If you also had the same initial thought as me, this is an excellent article - https://blog.eleuther.ai/transformer-math/ .
Re: Numbers every LLM Developer should know
#6I'm curious about the point on the embedding lookup cost... in my experience for an embedding lookup to be accurate, you have to include your entire document dataset to be queried against... obviously this can be just as expensive as querying a full cloud model if your dataset is very large. Interested if anyone had thoughts about this.
Re: Numbers every LLM Developer should know
#7This is honestly a bit gross, as it's just a marketing piece. The original numbers every programmer should know is a profound piece of pedagogy, aimed at helping programmers be better at their craft. This is just an excerpt from a pitch deck.
Where can I find the original?
Re: Numbers every LLM Developer should know
#8I'm curious about the point on the embedding lookup cost... in my experience for an embedding lookup to be accurate, you have to include your entire document dataset to be queried against... obviously this can be just as expensive as querying a full cloud model if your dataset is very large. Interested if anyone had thoughts about this.
There are very efficient algorithms for doing this, but of course it may still be expensive if your dataset is very large. See https://ann-benchmarks.com/ for some of the algorithms
Re: Numbers every LLM Developer should know
#9This is honestly a bit gross, as it's just a marketing piece. The original numbers every programmer should know is a profound piece of pedagogy, aimed at helping programmers be better at their craft. This is just an excerpt from a pitch deck.
Where can I find the original?
Its linked in the article: http://brenocon.com/dean_perf.html
Re: Numbers every LLM Developer should know
#10Actually, the only numbers every LLM developer should know are their accelerator specs.
For example:
A100 specs:
- 312e12 BF16 FLOPS
- 1555e9 GB/s HBM bandwidth
H100:
- 1000e12/2000e12 BF16/INT8 FLOPS
(apply ~0.7 flops efficiency multiplier because h100s power throttle extremely quickly)
- 3000 GB/s HBM bandwidth
---
For a 13B model on an A100, this nets:
13e9 * 2 bytes per param = 26 GB HBM required (at bf16)
26e9/1555e9 = 17ms / token small-batch latency (~60 tokens / second)
What about large batches?
latency for some batch size B is 13e9 * 2 FLOP per param * B / 312e12
We want B such that we're just about no longer HBM bound: 26e9/312e12 * B = 17ms
17e-3/(26e9/312e12)
giving a batch size of 204.
At that batch size (and all larger batch sizes), the a100 delivers a throughput of B * 1/17ms = 12000 tokens / second
---
KV caching, multi-gpu and -node comms and matmul efficiencies left as an exercise to the reader :)