> Running an LLM query through a GPU is very high latency: it may take, say, 5 seconds, with a throughput of 0.2 queries per second.
Why?
Numbers every LLM developer should know
81–90 of 107 posts
Re: Numbers every LLM developer should know
#82> 40-90%: Amount saved by appending “Be Concise” to your prompt
Looks to me like "numbers every LLM user needs to know".
Re: Numbers every LLM developer should know
#83Earlier quoted context omitted.
Vicuna is a delta model that you have to apply on top of LLaMA.
how does one get the original LLaMA weights. I tried the form that Meta has no dice. Also tried some torrents no luck there either
I filled the form and got the weights a few weeks later, but I work for a research organisation.
I thought the torrents were super active.
Re: Numbers every LLM developer should know
#84> LLM developer
This is the first time I heard this term, and when I Google search "LLM developer" in an incognito tab, different device, this article is one of the first results.
Seems like we should first establish what exactly is an LLM developer.
> When I was at Google, there was a document put together by Jeff Dean, the legendary engineer, called Numbers every Engineer should know.
The personal plug and appeal to authority of "When I was a Google" is unnecessary. "Numbers every Engineer should know" is public and literally linked there. It's a weird way to start a engineering blog post and makes it feel like marketing of one's resume. Then again, I guess that's what most of these engineering blog posts are nowadays.
Indeed Jeff Dean is a legend and needing to add the "legendary engineer" qualifier detracts from this point. Let these things speak for themselves.
Re: Numbers every LLM developer should know
#85Re: Numbers every LLM developer should know
#86> LLM developer This is the first time I heard this term, and when I Google search "LLM developer" in an incognito tab, different device, this article is one of the first results. Seems like we should first establish what exactly is an LLM developer. > When I was at Google, there was a document put together by Jeff Dean, the legendary engineer, called Numbers every Engineer should know. The personal plug and appeal t…
I think you're being somewhat uncharitable here. There's nothing wrong with adding a personal detail here or there, and nothing wrong with giving credit to those who deserve it. I don't see any reason to bikeshed the short, inessential details included in the blogger's prose.
Re: Numbers every LLM developer should know
#87> LLM developer This is the first time I heard this term, and when I Google search "LLM developer" in an incognito tab, different device, this article is one of the first results. Seems like we should first establish what exactly is an LLM developer. > When I was at Google, there was a document put together by Jeff Dean, the legendary engineer, called Numbers every Engineer should know. The personal plug and appeal t…
The term "LLM developer" is clear enough from context.
Re: Numbers every LLM developer should know
#88> 1.3: Average tokens per word
this is so US centric :-(
for billions of people, arguably the majority of the world, that’s incorrect
Re: Numbers every LLM developer should know
#89I’m confused. If I am an LLM developer why do I need to know the cost per token? That’s not the GPU cost, that’s a business decision from a company.
If I am an LLM user maybe that’s relevant but prone to being out of date. I’m not going to use this page as the source of truth on that anyways.
Since the article seems to be targeted at developers who use LLMs to e.g. generate Embeddings for semantic search, the title is about as accurate as saying a software engineer is a “keyboard developer” because they use a keyboard.
Re: Numbers every LLM developer should know
#90Earlier quoted context omitted.
> llama.cpp which runs a 13 billion parameter model on a 6GB GPU I think that's a typo there too, the 13B model needs like 10G of memory for 4 bits, it's the 7B one that fits into 6G. Well unless you do the split thing with some layers on the CPU I guess.
https://news.ycombinator.com/item?id=35937505
Yeah that's the split layer mode I mentioned. With 6G one can do about 18 layers, which is less than half of the 40 total for 13B.