Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 3.1

llama.meta.com

Post reply on HN
Print view View original

101–110 of 279 posts

Re: Llama 3.1

#101

Post by w4 »

monkmartinez wrote: post #60
jcmp wrote: post #36

"Meta AI isn't available yet in your country" Hi from europe :/

Why are (some) Europeans surprised when they are not included in tech product débuts? My lay understanding could best be described as; EU law is incredibly business unfriendly and takes a heroic effort in time and money to implement the myriad of requirements therein. Am I wrong?

> Why are (some) Europeans surprised when they are not included in tech product débuts?

We had a brief, abnormal, and special moment in time after the crypto wars ended in the mid-2000s where software products were truly global, and the internet was more or less unregulated and completely open (at least in most of the world). Sadly it seems that this era has come to a close, and people have not yet updated their understanding of the world to account for that fact.

People are also not great at thinking through the second order effects of the policies they advocate for (e.g. the GDPR), and are often surprised by the results.

Re: Llama 3.1

#102

Post by foundval »

quotemstr wrote: post #99
foundval wrote: post #74

You can chat with these new models at ultra-low latency at groq.com. 8B and 70B API access is available at console.groq.com. 405B API access for select customers only – GA and 3rd party speed benchmarks soon. If you want to learn more, there is a writeup at https://wow.groq.com/now-available-on-groq-the-largest-and-m... . (disclaimer, I am a Groq employee)

Groq's TSP architecture is one of the weirder and more wonderful ISAs I've seen lately. The choice of SRAM in fascinating. Are you guys planning on publishing anything about how you bridged the gap between your order-hundreds-megabytes SRAM TSP main memory and multi-TB model sizes?

There is a lot out here.

I gave a seminar about the overall approach recently, abstract: https://shorturl.at/E7TcA, recording: https://shorturl.at/zBcoL.

This two-part AMA has a lot more detail if you're already familiar with what we do:

https://www.youtube.com/watch?v=UztfweS-7MU

https://www.youtube.com/watch?v=GOGuSJe2C6U

Re: Llama 3.1

#103

Post by quotemstr »

foundval wrote: post #102
quotemstr wrote: post #99

Earlier quoted context omitted.

Groq's TSP architecture is one of the weirder and more wonderful ISAs I've seen lately. The choice of SRAM in fascinating. Are you guys planning on publishing anything about how you bridged the gap between your order-hundreds-megabytes SRAM TSP main memory and multi-TB model sizes?

There is a lot out here. I gave a seminar about the overall approach recently, abstract: https://shorturl.at/E7TcA , recording: https://shorturl.at/zBcoL . This two-part AMA has a lot more detail if you're already familiar with what we do: https://www.youtube.com/watch?v=UztfweS-7MU https://www.youtube.com/watch?v=GOGuSJe2C6U

Thanks!

Re: Llama 3.1

#104

Post by unraveller »

What are the substantial changes from 3.0 to 3.1 (70B) in terms of training approach? They don't seem to say how the training data differed just that both were 15T. I gather 3.0 was just a preview run and 3.1 was distilled down from the 405B somehow.

Re: Llama 3.1

#105

Post by Daunk »

monkmartinez wrote: post #60
jcmp wrote: post #36

"Meta AI isn't available yet in your country" Hi from europe :/

Why are (some) Europeans surprised when they are not included in tech product débuts? My lay understanding could best be described as; EU law is incredibly business unfriendly and takes a heroic effort in time and money to implement the myriad of requirements therein. Am I wrong?

Most things do dèbute in the EU, unless the product or company behind it doesn't value your privacy. Meta does not value your privacy.

Re: Llama 3.1

#107

Post by jermaustin1 »

beeboobaa3 wrote: post #76
jermaustin1 wrote: post #42

Earlier quoted context omitted.

You don't use the 405B parameter model at home. I have a lot of luck with 8B and 13B models on a single 3090. You can quantize them down (is that the term) which lowers precision and memory use, but still very usable... most of the time. If you are running a commercial service that uses AI, you buy a few dozen A100s, spend a half million, and you are good for a while. If you are running a commercial inferencing servi…

I can't expect all my users to have 3090s and if we're talking about spending millions there are better things to invest in than a stack of GPUs that will be obsolete in a year or three.

No, but if you are thinking about edge compute for LLMs, you quantize. Models are getting more efficient, and there are plenty of SLMs and smaller LLMs (like phi-2 or phi-3) that are plenty capable even on a tiny arm device like the current range of RPi "clones".

I have done experiments with 7B Llama3 Q8 models on a M3 MBP. They run faster than I can read, and only occasionally fall off the rails.

3B Phi-3 mini is almost instantaneous in simple responses on my MBP.

When I want longer context windows, I use a hosted service somewhere else, but if I only need 8000 tokens (99% of the time that is MORE than I need), any of my computers from the last 3 years are working just fine for it.

Re: Llama 3.1

#108

Post by dado3212 »

> We use synthetic data generation to produce the vast majority of our SFT examples, iterating multiple times to produce higher and higher quality synthetic data across all capabilities. Additionally, we invest in multiple data processing techniques to filter this synthetic data to the highest quality. This enables us to scale the amount of fine-tuning data across capabilities. [0]

Have other major models explicitly communicated that they're trained on synthetic data?

[0]. https://ai.meta.com/blog/meta-llama-3-1/

Re: Llama 3.1

#109

Post by angoragoats »

wuschel wrote: post #78
kingsleyopara wrote: post #68

Earlier quoted context omitted.

You might be able to get away with running a heavily quantizied 405b model using CPU inference at a blistering fast token every 5 seconds on a 7950x.

OK, I am curious now: What kind of hardware would I need to run such a model for a couple of users with decent performance? Where could I get a mapping of token / time vs hardware?

Unsure if anyone has specific hardware benchmarks for the 405b model yet, since it's so new, but elsewhere in this thread I outlined a build that'd probably be capable of running a quantized version of Llama 3.1 405b for roughly $10k.

The $10k figure is likely roughly the minimum amount of money/hardware that you'd need to run the model at acceptable speeds, as anything less requires you to compromise heavily on GPU cores (e.g. Tesla P40s also have 24GB of VRAM, for half the price or less, but are much slower than 3090s), or run on the CPU entirely, which I don't think will be viable for this model even with gobs of RAM and CPU cores, just due to its sheer size.

Re: Llama 3.1

#110

Post by stuckinhell »

cchance wrote: post #59
lelag wrote: post #58

The 405b model is actually competitive against closed source frontier models. Quick comparison with GPT-4o: +----------------+-------+-------+ | Metric | GPT-4o| Llama | | | | 3.1 | | | | 405B | +----------------+-------+-------+ | MMLU | 88.7 | 88.6 | | GPQA | 53.6 | 51.1 | | MATH | 76.6 | 73.8 | | HumanEval | 90.2 | 89.0 | | MGSM | 90.5 | 91.6 | +----------------+-------+-------+

Super cool, though sadly 405b will be outside most personal usage without cloud providers which sorta defeats the purpose of opensource to some extent atleast sadly, because .. nvidia's rampup of consumer VRAM is glacial

100% reddit is full of people trying to solder more vram
Post reply on HN