I wish you could download this without giving them your contact info
If only there were torrents…
Meta Llama 3
601–610 of 965 posts
Re: Meta Llama 3
#602Earlier quoted context omitted.
Not quite there yet, but very close and not done training! It's quite plausible that this model could be state of the art over GPT-4 in some domains when it finishes training, unless GPT-5 comes out first. Although 400B will be pretty much out of reach for any PC to run locally, it will still be exciting to have a GPT-4 level model in the open for research so people can try quantizing, pruning, distilling, and other…
The real question will be, how much you can quantize that while still retaining sanity. 400b at 2-bit would be possible to run on a Mac Studio - probably at multiple seconds per token, but sometimes that's "fast enough".
Yes. I expect an explosion of research and experimentation in model compression. The good news is I think there are tons of avenues that have barely been explored at all. We are at the very beginning of understanding this stuff, and my bet is that in a few years we'll be able to compress these models 10x or more.
Re: Meta Llama 3
#603Earlier quoted context omitted.
The day it's released, Llama-3-405B will be running on someone's Mac Studio. These models aren't that big. It'll be fine, just like Llama-2.
Maybe at 1 or 2 bits of quantization! Even the Macs with the most unified RAM are maxxed out with much smaller models than 405b (especially since it's a dense model and not a MOE).
[deleted]
Re: Meta Llama 3
#604They've got a console for it as well, https://www.meta.ai/ And announcing a lot of integration across the Meta product suite, https://about.fb.com/news/2024/04/meta-ai-assistant-built-wi... Neglected to include comparisons against GPT-4-Turbo or Claude Opus, so I guess it's far from being a frontier model. We'll see how it fares in the LLM Arena.
I'm based on LLaMA 2, which is a type of transformer language model developed by Meta AI. LLaMA 2 is a more advanced version of the original LLaMA model, with improved performance and capabilities. I'm a specific instance of LLaMA 2, trained on a massive dataset of text from the internet, books, and other sources, and fine-tuned for conversational AI applications. My knowledge cutoff is December 2022, and I'm constan…
Are you trying to say you are a bot?
Re: Meta Llama 3
#605Earlier quoted context omitted.
> It’s a bit like arguing that Linux is not open source because you don’t have every email Linus and the maintainers ever received. Or that you don’t know what lectures Linus attended or what books he’s read. Linux is open source, because you can actually compile it yourself! You don't need Linus's email for that (and if you needed some secret cryptographic key on Linus' laptop to decrypt and compile the kernel, then…
It's a blob that costs over $10,000,000 in electricity costs to compile. Even if they released everything only the rich could push go.
In today's Dwarkesh interview, Zuckerberg talks about energy becoming a limit for future models before cost or access to hardware does. Apparently current largest datacenters consume about 100MW, but Zuck is considering future ones consuming 1GW which is the output of typical nuclear reactor!
So, yeah, unless you own your own world-class datacenter, complete with the nuclear reactor necessary to power the training run, then training is not an option.
Re: Meta Llama 3
#606Earlier quoted context omitted.
For the still training 400B: Llama 3 GPT 4(Published) BBH 85.3 83.1 MMLU 86.1 86.4 DROP 83.5 80.9 GSM8K 94.1 92.0 MATH 57.8 52.9 HumEv 84.1 74.4 Although it should be noted that the API numbers were generally better than published numbers for GPT4. [1]: https://deepmind.google/technologies/gemini/
Hm, how much VRAM would this take to run?
"More than you can afford, pal--NVidia."
Re: Meta Llama 3
#607Earlier quoted context omitted.
> I just want to express how grateful I am that Zuck Praise for him at HN? It should be enough of a reason for him to pop a champagne today
Yeah, I'm also surprised at how many positive comments are in this thread. I do hate Facebook, but I also love engineers, so I'm not sure how to feel about this one.
I mean they basically invented, popularised and maintained react/react native which I've built my entire career on, I love them for that.
Re: Meta Llama 3
#608Earlier quoted context omitted.
His engineering mindset made him blind to the fact the metaverse was a product that nobody wanted or needed. In one of the Fridman interviews, he goes on and on about all the cool technical challenges involved in making the metaverse work. But when Fridman asked him what he likes to do in his spare time, it was all things that you could precisely not do in the metaverse. It was baffling to me that he failed to connec…
Let’s be honest VR is about the porn. I’d it’s successful at that Zuck will make his billions.
The computer game and television/movie industries both dwarf adult entertainment. The reasons for the rationale on how pornography made the VCR and VHS in particular a success (bringing affordable video pornography into the privacy of your home) do not apply to VR.
Re: Meta Llama 3
#609try to make it create a cube, a smooth cube. it doesn't seem to be able to, lots of little blocks but not a smooth single cube. curious
Re: Meta Llama 3
#610They've got a console for it as well, https://www.meta.ai/ And announcing a lot of integration across the Meta product suite, https://about.fb.com/news/2024/04/meta-ai-assistant-built-wi... Neglected to include comparisons against GPT-4-Turbo or Claude Opus, so I guess it's far from being a frontier model. We'll see how it fares in the LLM Arena.
Losers & Winners from Llama-3-400B Matching 'Claude 3 Opus' etc..
Losers:
- Nvidia Stock : lid on GPU growth in the coming year or two as "Nation states" use Llama-3/Llama-4 instead spending $$$ on GPU for own models, same goes with big corporations.
- OpenAI & Sam: hard to raise speculated $100 Billion, Given GPT-4/GPT-5 advances are visible now.
- Google : diminished AI superiority posture
Winners:
- AMD, intel: these companies can focus on Chips for AI Inference instead of falling behind Nvidia Training Superior GPUs
- Universities & rest of the world : can work on top of Llama-3