51–60 of 107 posts
RANDOM THOUGHT: i wonder when we are getting docker for llm ... a Modelfile ? FROM "PAAMA/16b" APPLY "MNO/DATASET" each layer could be lora adapter like thing maybe. maybe when AI chips are finally here.
https://pytorch.org/tutorials/beginner/pytorch_with_examples...
I think parts of the write-up are great. There are some unique assumptions being made in parts of the gist > 10: Cost Ratio of OpenAI embedding to Self-Hosted embedding > 1: Cost Ratio of Self-Hosted base vs fine-tuned model queries I don't know how useful these numbers are if you take away the assumptions that self-hosted will work as well as API. > 10x: Throughput improvement from batching LLM requests I see that t…
In a narrow use-case of a strict look-up. This seems to exaggerate the cost difference while having completely different trade-offs.
Earlier quoted context omitted.
The amount of get-off-my-lawn grognardness that LLM activity inspires is really ridiculous. I really would ask you to take a second look at the spirit of your comment and think carefully about how much you really understand about the work being done on top of LLMs and if it justifies this kind of response.
I had the same reaction as the OP. I’m not a data scientist by trade or title, but I would personally be a little offended. If you designed the Porsche 911, would you not be offended by the shade tree mechanic who simply knows how to change the oil calling himself a Porsche designer/engineer?
> Of course there are efforts to reduce this, notably llama.cpp which runs a 13 billion parameter model on a 6GB GPU by quantizing aggressively down to 4 bits (and 8 bits without too much impact), but that’s atypical. No, 4bit quantization is the typical case. At 4bit you can fit twice the parameters of 8bit in the same space for far better performance/perplexity/quality. Running LLMs higher than 4bit is atypical and…
[Author] Completely disagree. Any analysis shows that you see perplexity reduction at 4 bits. Have a look at llama.cpp's results here: https://github.com/ggerganov/llama.cpp#quantization 4 bit has a perplexity score 0.13 or so higher.
> ~$1 million: Cost to train a 13 billion parameter model on 1.4 trillion tokens MosaicML claims they trained a 7 billion parameter on 1 trillion tokens with a budget of $200k. https://www.mosaicml.com/blog/mpt-7b Does training cost scale linearly with model size and token count? If so, that suggests a lower bound of $600k to train the 13 billion params model. (Still roughly the same magnitude)
[Author] Mosaic must be getting some kind of sweetheart deals on A100 80GB and A100 40GB. The prices they are quoting are not what say the AWS on-demand prices are. They quote $2 per GPU for A100 40GB and $2.50 for A100 80GB. That's literally half the AWS on-demand rate for A100s here: https://aws.amazon.com/ec2/instance-types/p4/ And these are impossible to get. We tried to get some for Anyscale, and we were told th…
Llama paper mentioned 135,168 A100 hours for training 13 billion model on 1 trillion tokens, which means ~$150k for lambdalabs on demand instance.
> ~$1 million: Cost to train a 13 billion parameter model on 1.4 trillion tokens Llama paper mentioned 135,168 A100 hours for training 13 billion model on 1 trillion tokens, which means ~$150k for lambdalabs on demand instance.
Plus they don't actually have any actually A100s available at the moment (2022-05-17).
CoreWeave is a nice middle ground. You can at least get the A100 machines into a k8s cluster.
> There’s usually no need to go beyond 16-bit accuracy, and most of the time when you go to 8-bit accuracy there is too much loss of resolution. I'm not sure this is accurate. From what I have seen, 8-bit quantization is usually fine, and even 4-bit is a viable tradeoff. Here are some benchmarks from TextSynth showing no significant degradation between 16 and 8 bit: https://textsynth.com/technology.html 8-bit uses ha…
I'm also confused about this:
> ~$1 million: Cost to train a 13 billion parameter model on 1.4 trillion tokens
This is apparently related to the LLaMa paper, but that paper seems to cite 1.0T tokens (rather than 1.4T tokens) for the 13B model. Also, if 20 to 1 is in fact optimal for the data-to-parameter ratio, then using a 100 to 1 ratio doesn't seem like an appropriate way to arrive at a magic number for training costs. The magic number should really be based on an optimal configuration. Or, perhaps, my superficial understanding here leads me to miss some important distinctions.
I would add the following two numbers if you're generating realtime text or speech for human consumption: - Human Reading Speed (English): ~250 words per minute - Human Speaking Speed (English): ~150 words per minute Should be treated like the Doherty Threshold [1] for generative content. [1] https://lawsofux.com/doherty-threshold/
But I'd say LLMs produce content faster than I can read or write it, because they can produce content which is really dense. Ask GPT-4 a question and then answer it yourself. Maybe your answer will be as good or better than GPT-4's but GPT-4 writes its answer a lot faster.