Llama 3.1 • Hacker News Board

Re: Llama 3.1

#171

Post by Aurornis » Tue, Jul 23, 2024, 9:34 PM UTC

cchance wrote: post #59

lelag wrote: post #58

The 405b model is actually competitive against closed source frontier models. Quick comparison with GPT-4o: +----------------+-------+-------+ | Metric | GPT-4o| Llama | | | | 3.1 | | | | 405B | +----------------+-------+-------+ | MMLU | 88.7 | 88.6 | | GPQA | 53.6 | 51.1 | | MATH | 76.6 | 73.8 | | HumanEval | 90.2 | 89.0 | | MGSM | 90.5 | 91.6 | +----------------+-------+-------+

Super cool, though sadly 405b will be outside most personal usage without cloud providers which sorta defeats the purpose of opensource to some extent atleast sadly, because .. nvidia's rampup of consumer VRAM is glacial

> sorta defeats the purpose of opensource to some extent

Not in the slightest. They even have a table of cloud providers where you can host the 405B model and the associated cost to do so on their website: https://llama.meta.com/ (Scroll down)

"Open Source" doesn't mean "You can run this on consumer hardware". It just means that it's open source. They also released 8B and 70B models for people to use on consumer gear.

Re: Llama 3.1

#172

Post by ajhai » Tue, Jul 23, 2024, 9:38 PM UTC

jxy wrote: post #167

ajhai wrote: post #124

You can already run these models locally with Ollama (ollama run llama3.1:latest) along with at places like huggingface, groq etc. If you want a playground to test this model locally or want to quickly build some applications with it, you can try LLMStack ( https://github.com/trypromptly/LLMStack ). I wrote last week about how to configure and use Ollama with LLMStack at https://docs.trypromptly.com/guides/using-llam…

You are a maintainer of a software that depends on ollama, so you should know that ollama depends on llama.cpp. And as of now, llama.cpp doesn't support the new ROPE: https://github.com/ggerganov/llama.cpp/issues/8650 , and all ollama can do is wait for llama.cpp: https://github.com/ollama/ollama/issues/5881

I've tested Q4 on M1 and it works though the quality may not likely be the same as you'd expect as others have pointed out on the issue.

Re: Llama 3.1

#173

Post by tymscar » Tue, Jul 23, 2024, 9:42 PM UTC

Sakthimm wrote: post #150

lawlessone wrote: post #10

Earlier quoted context omitted.

maybe someone will figure out some ways to prune/ quantize it a huge amount ;-; edit: If the AI bubble pops we will be swimming in GPUs... but no new models.

This is absurd. We have crossed the point of no return, llms will forever be in our lives in one form or another, just like internet, especially with the release of these open model weights. There is no bubble, only way forward is better, efficient llms, everywhere.

You seem to not understand what a bubble popping is. Yes we have the internet around, that doesn’t mean the dot com bubble didn’t pop…

Re: Llama 3.1

#174

Post by xu_ituairo » Tue, Jul 23, 2024, 9:49 PM UTC

nomel wrote: post #166

fngjdflmdflg wrote: post #160

Earlier quoted context omitted.

As far as I know it's not just the weights. it's everything but the dataset. So the code used to generate the weights is also open source.

Is there any other case where "open source" is used for something that can't be reproduced? Seems like a new term is required, in the concept of "open source, non-reproducible artifacts". I suppose language changes. I just prefer it changes towards being more precise, not less.

This feels somewhat analogous to games like Quake being open-sourced though still needing the user to provide the original game data files.

Re: Llama 3.1

#175

Post by buu700 » Tue, Jul 23, 2024, 9:49 PM UTC

hmottestad wrote: post #170

buu700 wrote: post #161

Earlier quoted context omitted.

I don't have the hardware to confirm this, so I'd take it with a grain of salt, but ChatGPT tells me that a maxed out M3 MacBook Pro with 128 GB RAM should be capable of efficiently running Llama 3.1 405B, albeit with essentially no ability to multitask. (It also predicted that a MacBook Air in 2030 will be able to do the same, and that for smartphones to do the same might take around 20 years.)

I’ve run the Falcon 180B on my M3 Max with 128 GB of memory. I think I ran it at 3-bit. Took a long time to load and was incredibly slow at generating text. Even if you could load the Llama 405B model it would be too slow to be of much use.

Ah, that's a shame to hear. FWIW, ChatGPT did also suggest that there was a lot of room for improvement in the MPS backend of PyTorch that would likely make it more efficient on Apple hardware in time.

Re: Llama 3.1

#176

Post by crimsoneer » Tue, Jul 23, 2024, 9:55 PM UTC

monkmartinez wrote: post #60

jcmp wrote: post #36

"Meta AI isn't available yet in your country" Hi from europe :/

Why are (some) Europeans surprised when they are not included in tech product débuts? My lay understanding could best be described as; EU law is incredibly business unfriendly and takes a heroic effort in time and money to implement the myriad of requirements therein. Am I wrong?

You are pretty wrong. EU law is tricky on AI very specifically in this use case (because it's a massive model), but that's not affecting anybody else.

Other than that, and GDPR (which is generally now regarded as a good thing), I'm not sure what requirements you've got in mind.

Re: Llama 3.1

#177

Post by Klaus23 » Tue, Jul 23, 2024, 10:39 PM UTC

buu700 wrote: post #175

hmottestad wrote: post #170

Earlier quoted context omitted.

I’ve run the Falcon 180B on my M3 Max with 128 GB of memory. I think I ran it at 3-bit. Took a long time to load and was incredibly slow at generating text. Even if you could load the Llama 405B model it would be too slow to be of much use.

Ah, that's a shame to hear. FWIW, ChatGPT did also suggest that there was a lot of room for improvement in the MPS backend of PyTorch that would likely make it more efficient on Apple hardware in time.

You fundamentally misunderstand the bottleneck of large LLMs. It is not really possible to make gains that way.

A 405B LLM has 405 billion parameters. If you run it at full "prescision", each parameter takes up 2 bytes, which means you need 810GB of memory. If it does not fit in RAM or GPU memory it will swap to disc and be unusably slow.

You can run the model at reduced prescision to save memory, called quantisation, but this will degrade the quality of the response. The exact amount of degradation depends on the task, the specific model and its size. Larger models seem to suffer slightly less. 1 byte per parameter is pretty much as good as full precision. 4 bits per parameter is still good quality, 3 bits is noticeably worse and 2 bits is often bad to unusable.

With 128GB of RAM, zero overhead and a 405B model, you would have to quantize to about 2.5 bits, which would noticeably degrade the response quality.

There is also model pruning, which removes parameters completely, but this is much more experimental than quantisation, also degrades response quality, and I have not seen it used that widely.

Re: Llama 3.1

#178

Post by diego_sandoval » Tue, Jul 23, 2024, 10:57 PM UTC

cchance wrote: post #59

lelag wrote: post #58

The 405b model is actually competitive against closed source frontier models. Quick comparison with GPT-4o: +----------------+-------+-------+ | Metric | GPT-4o| Llama | | | | 3.1 | | | | 405B | +----------------+-------+-------+ | MMLU | 88.7 | 88.6 | | GPQA | 53.6 | 51.1 | | MATH | 76.6 | 73.8 | | HumanEval | 90.2 | 89.0 | | MGSM | 90.5 | 91.6 | +----------------+-------+-------+

Super cool, though sadly 405b will be outside most personal usage without cloud providers which sorta defeats the purpose of opensource to some extent atleast sadly, because .. nvidia's rampup of consumer VRAM is glacial

The fact that it takes $20k to run your own SOTA model, instead of the $2B+ that it took until yesterday, is significant.

Re: Llama 3.1

#179

Post by sebastiennight » Tue, Jul 23, 2024, 11:03 PM UTC

sangnoir wrote: post #88

sfblah wrote: post #73

Is there an actual open-source community around this in the spirit of other ones where people outside meta can somehow "contribute" to it? If I wanted to "work on" this somehow, what would I do?

There are a bunch of downstream fine-tuned and/or quantized models where people collaborate and share their recipes. In terms of contributing to Llama itself - I suspect Meta wants (or needs) code contributions at this time.

Did you mean, Meta does not want or need code contributions? It would seem to make more sense.

Re: Llama 3.1

#180

Post by rcarmo » Tue, Jul 23, 2024, 11:03 PM UTC

jxy wrote: post #169

rcarmo wrote: post #148

Working great in ollama: https://mastodon.social/@rcarmo/112837520236956526

https://github.com/ollama/ollama/issues/5881 https://github.com/ggerganov/llama.cpp/issues/8650

Still works fine for me. Latest ollama, running on NVIDIA.