Earlier quoted context omitted.
Two 128gb Mac studios networked via thunderbolt 4?
This is actually a promising endeavor. Id love to see someone try that.
There's already at least one project that attempts this:
Llama 3.1
91–100 of 279 posts
Re: Llama 3.1
#92What kind of machine do I need to run 405B local?
You can't. Sorry. Unless... You have a couple hundred $k sitting around collecting dust... then all you need is a DGX or HGX level of vRAM, the power to run it, the power to keep it cool, and place for it to sit.
You could run a 4bit quant for about $10k I'm guessing. 10x3090s would do.
Re: Llama 3.1
#93I wrote about this when llama-3 came out, and this launch confirms it: Meta's goal from the start was to target OpenAI and the other proprietary model players with a "scorched earth" approach by releasing powerful open models to disrupt the competitive landscape. Meta can likely outspend any other AI lab on compute and talent: - OpenAI makes an estimated revenue of $2B and is likely unprofitable. Meta generated a rev…
> Open source likely attracts better talent and researchers I work at OpenAI and used to work at meta. Almost every person from meta that I know has asked me for a referral to OpenAI. I don’t know anyone who left OpenAI to go to meta.
What % of them were from FAIR vs non-FAIR?
Re: Llama 3.1
#94You can chat with these new models at ultra-low latency at groq.com. 8B and 70B API access is available at console.groq.com. 405B API access for select customers only – GA and 3rd party speed benchmarks soon. If you want to learn more, there is a writeup at https://wow.groq.com/now-available-on-groq-the-largest-and-m... . (disclaimer, I am a Groq employee)
We also added Llama 3.1 405B to our VSCode copilot extension for anyone to try coding with it.
Free trial gets you 50 messages, no credit card required - https://double.bot
(disclaimer, I am the co-founder)
Re: Llama 3.1
#95The 405b model is actually competitive against closed source frontier models. Quick comparison with GPT-4o: +----------------+-------+-------+ | Metric | GPT-4o| Llama | | | | 3.1 | | | | 405B | +----------------+-------+-------+ | MMLU | 88.7 | 88.6 | | GPQA | 53.6 | 51.1 | | MATH | 76.6 | 73.8 | | HumanEval | 90.2 | 89.0 | | MGSM | 90.5 | 91.6 | +----------------+-------+-------+
Super cool, though sadly 405b will be outside most personal usage without cloud providers which sorta defeats the purpose of opensource to some extent atleast sadly, because .. nvidia's rampup of consumer VRAM is glacial
Great for Groq whos already hosting it but at what cost I guess.
Re: Llama 3.1
#96Wow! The benchmarks are truly impressive, showing significant improvements across almost all categories. It's fascinating to see how rapidly this field is evolving. If someone had told me last year that Meta would be leading the charge in open-source models, I probably wouldn't have believed them. Yet here we are, witnessing Meta's substantial contributions to AI research and democratization.
On a related note, for those interested in experimenting with large language models locally, I've been working on an app called Msty [1]. It allows you to run models like this with just one click and features a clean, functional interface. Just added support for both 8B and 70B. Still in development, but I'd appreciate any feedback.
[1]: https://msty.app
Re: Llama 3.1
#97You can chat with these new models at ultra-low latency at groq.com. 8B and 70B API access is available at console.groq.com. 405B API access for select customers only – GA and 3rd party speed benchmarks soon. If you want to learn more, there is a writeup at https://wow.groq.com/now-available-on-groq-the-largest-and-m... . (disclaimer, I am a Groq employee)
405B is already being served on WhatsApp! https://ibb.co/kQ2tKX5
How do you get that option?
Re: Llama 3.1
#98What kind of machine do I need to run 405B local?
You can't. Sorry. Unless... You have a couple hundred $k sitting around collecting dust... then all you need is a DGX or HGX level of vRAM, the power to run it, the power to keep it cool, and place for it to sit.
You can build a machine that will run the 405b model for much, much less, if you're willing to accept the following caveats:
* You'll be running a Q5(ish) quantized model, not the full model
* You're OK with buying used hardware
* You have two separate 120v circuits available to plug it into (I assume you're in the US), or alternatively a single 240v dryer/oven/RV-style plug.
The build would look something like (approximate secondary market prices in parentheses):
* Asrock ROMED8-2T motherboard ($700)
* A used Epyc Rome CPU ($300-$1000 depending on how many cores you want)
* 256GB of DDR4, 8x 32GB modules ($550)
* nvme boot drive ($100)
* Ten RTX 3090 cards ($700 each, $7000 total)
* Two 1500 watt power supplies. One will power the mobo and four GPUs, and the other will power the remaining six GPUs ($500 total)
* An open frame case, the kind made for crypto miners ($100?)
* PCIe splitters, cables, screws, fans, other misc parts ($500)
Total is about $10k, give or take. You'll be limiting the GPUs (using `nvidia-smi` or similar) to run at 200-225W each, which drastically reduces their top-end power draw for a minimal drop in performance. Plug each power supply into a different AC circuit, or use a dual 120V adapter with a 240V outlet to effectively accomplish the same thing.
When actively running inference you'll likely be pulling ~2500-2800W from the wall, but at idle, the whole system should use about a tenth of that.
It will heat up the room it's in, especially if you use it frequently, but since it's in an open frame case there are lots of options for cooling.
I realize that this setup is still out of the reach of the "average Joe" but for a dedicated (high-end) hobbyist or someone who wants to build a business, this is a surprisingly reasonable cost.
Edit: the other cool thing is that if you use fast DDR4 and populate all 8 RAM slots as I recommend above, the memory bandwidth of this system is competitive with that of Apple silicon -- 204.8GB/sec, with DDR4-3200. Combined with a 32+ core Epyc, you could experiment with running many models completely on the CPU, though Lllama 405b will probably still be excruciatingly slow.
Re: Llama 3.1
#99You can chat with these new models at ultra-low latency at groq.com. 8B and 70B API access is available at console.groq.com. 405B API access for select customers only – GA and 3rd party speed benchmarks soon. If you want to learn more, there is a writeup at https://wow.groq.com/now-available-on-groq-the-largest-and-m... . (disclaimer, I am a Groq employee)
Groq's TSP architecture is one of the weirder and more wonderful ISAs I've seen lately. The choice of SRAM in fascinating. Are you guys planning on publishing anything about how you bridged the gap between your order-hundreds-megabytes SRAM TSP main memory and multi-TB model sizes?
Re: Llama 3.1
#100Related:
Open Source AI Is the Path Forward
https://about.fb.com/news/2024/07/open-source-ai-is-the-path...