What kind of machine do I need to run 405B local?
according to another comment, ~10x 4090 video cards.
That was the punchline of a joke.
Llama 3.1
81–90 of 279 posts
Re: Llama 3.1
#82The 405B model is already being served on WhatsApp:
https://ibb.co/kQ2tKX5
Re: Llama 3.1
#83You can chat with these new models at ultra-low latency at groq.com. 8B and 70B API access is available at console.groq.com. 405B API access for select customers only – GA and 3rd party speed benchmarks soon. If you want to learn more, there is a writeup at https://wow.groq.com/now-available-on-groq-the-largest-and-m... . (disclaimer, I am a Groq employee)
405B is already being served on WhatsApp!
Re: Llama 3.1
#84Re: Llama 3.1
#85Earlier quoted context omitted.
Your going to need a lot more than a few, 800G VRAM needed
how is this even useful? no one can run it.
If you want to run the 405B model without spending thousands of dollars on dedicated hardware, you rent compute from a datacenter. Meta lists AWS, Google and Microsoft among others as cloud partners.
But also check out the 8B and 70B Llama-3.1 models which show improved benchmarks over the Llama-3 models released in April.
Re: Llama 3.1
#86Earlier quoted context omitted.
If previous quantization results hold up, fp8 will have nearly identical performance while using 405GiB for weights, but the KV cache size will still be significant. Too bad, too, I don't think my PC will fit 20 4090s (480GiB).
I've got a motherboard that will support 8!
40,320 4090s?? What witchcraft is this?! :D
Re: Llama 3.1
#87Are there any other models with free unlimited use like chatgpt?
mistral.ai
Re: Llama 3.1
#88Is there an actual open-source community around this in the spirit of other ones where people outside meta can somehow "contribute" to it? If I wanted to "work on" this somehow, what would I do?
There are a bunch of downstream fine-tuned and/or quantized models where people collaborate and share their recipes. In terms of contributing to Llama itself - I suspect Meta wants (or needs) code contributions at this time.
Re: Llama 3.1
#89Earlier quoted context omitted.
This is very impressive, though an adjacent question — does anyone know roughly how much time and compute cost it takes to train something like the 405B? I would imagine with all the compute Meta has that the moat is incredibly large in terms of being able to train multiple 405B-level morels and compete.
30.84M H100 compute-hours, according to the model card https://github.com/meta-llama/llama-models/blob/main/models/...
Interestingly that’s less energy than the mass energy equivalent of one gram of matter or roughly 5 seconds worth of the worlds average energy consumption (according to wolfram alpha). Still an absolute insane amount of energy, as in about 5 million dollars at household electricity rates. Absolutely wild how much compute goes into this.
Re: Llama 3.1
#90The 405b model is actually competitive against closed source frontier models. Quick comparison with GPT-4o: +----------------+-------+-------+ | Metric | GPT-4o| Llama | | | | 3.1 | | | | 405B | +----------------+-------+-------+ | MMLU | 88.7 | 88.6 | | GPQA | 53.6 | 51.1 | | MATH | 76.6 | 73.8 | | HumanEval | 90.2 | 89.0 | | MGSM | 90.5 | 91.6 | +----------------+-------+-------+
Super cool, though sadly 405b will be outside most personal usage without cloud providers which sorta defeats the purpose of opensource to some extent atleast sadly, because .. nvidia's rampup of consumer VRAM is glacial
I agree that 405B isn't practical for home users, but I disagree that it defeats the purpose of open source. If you're building a business on inference it can be valuable to run an open model on hardware that you control, without the need to worry that OpenAI or Anthropic or whoever will make drastic changes to the model performance or pricing. Also, it allows the possibility of fine-tuning the model to your requirements. Meta believes it's in their interest to promote these businesses.
I'd think of the 405B model as the equivalent to a big rig tractor trailer. It's not for home use. But also check out the benchmark improvements for the 70B and 8B models.