Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 3.1

llama.meta.com

Post reply on HN
Print view View original

161–170 of 279 posts

Re: Llama 3.1

#161

Post by buu700 »

api wrote: post #146
paxys wrote: post #137

Earlier quoted context omitted.

You don't need a model of this scale for personal use. Llama 3.1 8B can easily run on your laptop right now. The 70B model can run on a pair of 4090s.

I have the 70b model running quantized just fine on an M1 Max laptop with 64GiB unified RAM. Performance is fine and so far some Q&A tests are impressive. This is good enough for a lot of use cases... on a laptop. An expensive laptop, but hardware only gets better and cheaper over time.

I don't have the hardware to confirm this, so I'd take it with a grain of salt, but ChatGPT tells me that a maxed out M3 MacBook Pro with 128 GB RAM should be capable of efficiently running Llama 3.1 405B, albeit with essentially no ability to multitask.

(It also predicted that a MacBook Air in 2030 will be able to do the same, and that for smartphones to do the same might take around 20 years.)

Re: Llama 3.1

#162

Post by michaelt »

drdaeman wrote: post #122
michaelt wrote: post #71

Earlier quoted context omitted.

I suspect 30B models are in a weird spot, too big for widespread home use, too small for cutting edge performance. For home users 7B models (which can fit on an 8GB GPU) and 13B models (which can fit on a 16GB GPU) are in far more demand. If you're a researcher, you want a 70B model to get the best performance, and so your benchmarks are comparable to everyone else.

I thought home use is whatever fits in 24GB (a single 3090 GPU, which is pretty affordable), not 8 or 16. 30B models fit.

While some home users do indeed have 24GB of vram, the fact is a 4090 costs $1700

Such models will never top the number of downloads charts, or the community hype, as there’s just loads more people who can use the smaller models.

And if you can afford one 4090 you can probably afford two.

Re: Llama 3.1

#164

Post by j_maffe »

ninjin wrote: post #155
moffkalast wrote: post #147

Earlier quoted context omitted.

> specifically, it puts restrictions on commercial use for some users (paragraph 2) and also restricts the use of the model and software for certain purposes (the Acceptable Use Policy) It's "a Google and Apple can't use this model in production" clause that frankly we can all be relatively okay with.

Good, then we can expect them to call it what it is then? Not open source and not open science and a regression in terms of openness in relationship to what came before. Because that is precisely my objection. There are those of us that have been committed to those ideals for a long time and now one of the largest corporations on earth is appropriating those terms for marketing purposes.

I think it's great that you're fighting to maintain the term's fundamental meaning. I do, however, think that we need to give credit where credit is due to companies who take actions in the right direction to encourage more companies to do the same. If we blindly protest any positive-impact action by corporations for not being perfect, they'll get the hint and stop trying to appease the community entirely.

Re: Llama 3.1

#165

Post by coconut08 »

microtonal wrote: post #132
wuschel wrote: post #78

Earlier quoted context omitted.

OK, I am curious now: What kind of hardware would I need to run such a model for a couple of users with decent performance? Where could I get a mapping of token / time vs hardware?

You can run the 4-bit GPTQ/AWQ quantized Llama 405B somewhat reasonably on 4x H100 or A100. You will be somewhat limited in how many tokens you can have in flight between requests and you cannot create CUDA graphs for larger batch sizes. You can run 405B well on 8x H100 and A100, either with the mixed BFloat16/FP8 checkpoint that Meta provided or GPTQ/AWQ-quantized models. Note though that the A100 does not have nati…

how much vram do you need for 4-bit llama 405?

Re: Llama 3.1

#166

Post by nomel »

fngjdflmdflg wrote: post #160
nomel wrote: post #154

Earlier quoted context omitted.

I really wish people would use "open weights" rather than "open source". It's precise and obvious, and leaves an accurate descriptor for actual "open source" models, where the source and methods that that generate the artifact , that is the weights, is open.

As far as I know it's not just the weights. it's everything but the dataset. So the code used to generate the weights is also open source.

Is there any other case where "open source" is used for something that can't be reproduced? Seems like a new term is required, in the concept of "open source, non-reproducible artifacts".

I suppose language changes. I just prefer it changes towards being more precise, not less.

Re: Llama 3.1

#167

Post by jxy »

ajhai wrote: post #124

You can already run these models locally with Ollama (ollama run llama3.1:latest) along with at places like huggingface, groq etc. If you want a playground to test this model locally or want to quickly build some applications with it, you can try LLMStack ( https://github.com/trypromptly/LLMStack ). I wrote last week about how to configure and use Ollama with LLMStack at https://docs.trypromptly.com/guides/using-llam…

You are a maintainer of a software that depends on ollama, so you should know that ollama depends on llama.cpp. And as of now, llama.cpp doesn't support the new ROPE: https://github.com/ggerganov/llama.cpp/issues/8650, and all ollama can do is wait for llama.cpp: https://github.com/ollama/ollama/issues/5881

Re: Llama 3.1

#168

Post by handzhiev »

AaronFriel wrote: post #43

Is there pricing available on any of these vendors? Open source models are very exciting for self hosting, but the per-token hosted inference pricing hasn't been competitive with OpenAI and Anthropic, at least for a given tier of quality. (E.g.: Llama 3 70B costing between $1 and $10 per million tokens on various platforms, but Claude Sonnet 3.5 is $3 per million.)

Llama 3 is 0.59/0.79 on Groq. Still no price for 3.1

Re: Llama 3.1

#170

Post by hmottestad »

buu700 wrote: post #161
api wrote: post #146

Earlier quoted context omitted.

I have the 70b model running quantized just fine on an M1 Max laptop with 64GiB unified RAM. Performance is fine and so far some Q&A tests are impressive. This is good enough for a lot of use cases... on a laptop. An expensive laptop, but hardware only gets better and cheaper over time.

I don't have the hardware to confirm this, so I'd take it with a grain of salt, but ChatGPT tells me that a maxed out M3 MacBook Pro with 128 GB RAM should be capable of efficiently running Llama 3.1 405B, albeit with essentially no ability to multitask. (It also predicted that a MacBook Air in 2030 will be able to do the same, and that for smartphones to do the same might take around 20 years.)

I’ve run the Falcon 180B on my M3 Max with 128 GB of memory. I think I ran it at 3-bit. Took a long time to load and was incredibly slow at generating text. Even if you could load the Llama 405B model it would be too slow to be of much use.
Post reply on HN