Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI releases Code Llama 70B

twitter.com

Post reply on HN
Print view View original

131–140 of 311 posts

Re: Meta AI releases Code Llama 70B

#131

Post by nullstyle »

sbrother wrote: post #109
coder543 wrote: post #67

Earlier quoted context omitted.

Quantization is highly effective at reducing memory and storage requirements, and it barely has any impact on quality (unless you take it to the extreme). Approximately no one should ever be running the full fat fp16 models during inference of any of these LLMs. That would be incredibly inefficient. I run 33B parameter models on my RTX 3090 (24GB VRAM) no problem. 70B should easily fit into 64GB of RAM.

Can I ask how many tok/s you're getting on that setup? I'm trying to decide whether to invest in a high-end NVIDIA setup or a Mac Studio with llama.cpp for the purposes of running LLMs like this one locally.

Here's an example of megadolphin running on my m2 ultra setup: https://gist.github.com/nullstyle/a9b68991128fd4be84ffe8435f...

Re: Meta AI releases Code Llama 70B

#132

Post by colesantiago »

Philpax wrote: post #91
colesantiago wrote: post #57

Earlier quoted context omitted.

Everyone using AI in production is using Pytorch by Meta. Which is open source. I do not know anybody important in the AI space apart from Google using TensorFlow.

That may be true, but it's largely irrelevant. The ML framework in use has no bearing on whether or not you have the data required to reproduce the model being trained with that framework.

Do you and the GP have 350K GPUs and quality data to reproduce 1:1 whatever Facebook releases in their repos?

Even if you want to reproduce the model and they give you the data, you would need to do this at Facebook scale, so you and the GP are just making moot points all around.

https://about.fb.com/news/2023/05/metas-infrastructure-for-a...

https://www.theregister.com/2024/01/20/metas_ai_plans/

The fact that these models are coming from Meta in the open rather than Google which releases only papers with no model tell's me that Meta's models is open enough for everyone to use.

Besides, everyone using the Pytorch framework benefits Meta in the same way they were originally founded as a company:

Network effects

It's relevant.

Re: Meta AI releases Code Llama 70B

#133

Post by FrustratedMonky »

svara wrote: post #61
Havoc wrote: post #23

Not sure who this is aimed at? The avg programmer probably doesn’t have the gear on hand to run this at the required pace Cool nonetheless

It's aimed at OpenAI's moat. Making sure they don't accumulate too much of one. No one actually has to use this, it just needs to be clear that LLM as a service won't be super high margin because competition can simply start building on Meta's open source releases.

So. Strange as it seems, is Meta being more 'Open', than OpenAI that was created to be the 'open' option to fight off Meta and Google?

Re: Meta AI releases Code Llama 70B

#134

Post by MeImCounting »

reddit_clone wrote: post #121
moyix wrote: post #59

Earlier quoted context omitted.

You can run it on a Macbook M1/M2 with 64GB of RAM.

I am not too familiar with LLMs and GPUs (Not a gamer either). But want to learn. Could you please expand on what else would be capable of running such models locally? How about a linux laptop/desktop with specific hardware configuration?

It pretty much comes down to 2 factors which is memory bandwidth and compute. You need a high enough memory bandwidth to be able to "feed" the compute and you need beefy enough compute to be able to keep up with the data that is being fed in by the memory. In theory a single Nvidia 4090 would be able to run a 70b model with quantization at "useable" speeds. The reason mac hardware is so capable in AI is because of the unified architecture meaning the memory is shared across the GPU and CPU. There are other factors but it essentially comes down to tokens per second advantages. You could run one of these models on an old GPU with low memory bandwidth just fine but your tokens per second would be far too slow for what most people consider "useable" and the quantization necessary might star noticeably effecting the quality.

Re: Meta AI releases Code Llama 70B

#135

Post by flir »

bk146 wrote: post #70

Can someone explain Meta's strategy with the open source models here? Genuine question, I don't fully undestand. (Please don't say "commoditize your complement" without explaining what exactly they're commoditizing...)

Really enjoying how many different answers you got.

(My theory: if there's an AI pot of gold, what megacorp can risk one of the others getting to it first?)

Re: Meta AI releases Code Llama 70B

#136

Post by Too »

bk146 wrote: post #70

Can someone explain Meta's strategy with the open source models here? Genuine question, I don't fully undestand. (Please don't say "commoditize your complement" without explaining what exactly they're commoditizing...)

Meta still sit on all the juicy user data that they want to use AI on but they don’t know how. They are crowdsourcing development of applications and tooling.

Meta releases model. Joe builds a cool app with it, earns some internet points and if lucky a few hundred bucks. Meta copies app, multiply Joes success story with 1 billion users and earn a few million bucks.

Joe is happy, Meta is happy. Everybody is happy.

Re: Meta AI releases Code Llama 70B

#137

Post by eurekin »

kuczmama wrote: post #130
nullc wrote: post #122

Earlier quoted context omitted.

You're not required to put the whole model in a single GPU. You can buy a 24GB gpu for $150-ish (P40).

Wow that's a really good idea. I could potentially buy 4 Nvidia P40's for the same price as a 3090 and run inference on pretty much any model I want.

Just make sure you're comfortable with manually compiling the bitsandbytes and generally combine a software stack of almost out of date libraries

Re: Meta AI releases Code Llama 70B

#138

Post by importantbrian »

observationist wrote: post #97
eurekin wrote: post #75

Earlier quoted context omitted.

Total speculation: Yann LeCun is there and he is really passionate about the technology and openness

The faux-open models mean the models can't be used in competing products. The open code base means enthusiasts and amateurs and other people hack on Meta projects and contribute improvements. They get free R&D and suppress competition, while looking like they have principles. Yann is clueless about open source principles, or the models would have been Apache or some other comparably open license. It's all ruthless co…

Meta's choice of license doesn't indicate that Yann is clueless about open-source principles. I don't know about Meta specifically, but in most companies choosing the license for open source projects involves working with a lot of different stakeholders. He very easily could have pushed for Apache or MIT and some other interest group within Meta vetoed it.

Re: Meta AI releases Code Llama 70B

#139

Post by adventured »

svara wrote: post #61
Havoc wrote: post #23

Not sure who this is aimed at? The avg programmer probably doesn’t have the gear on hand to run this at the required pace Cool nonetheless

It's aimed at OpenAI's moat. Making sure they don't accumulate too much of one. No one actually has to use this, it just needs to be clear that LLM as a service won't be super high margin because competition can simply start building on Meta's open source releases.

The moat is all but guaranteed to be the scale of the GPUs required to operate these for a lot of users as they get ever larger, specifically the extreme cost that is going along with that.

Anybody have $10 billion sitting around to deploy that gigantic open source set-up for millions of users? There's your moat and only a relatively few companies will be able to do it.

One of Google's moats is, has been, and will always be the scale required to just get into the search game and the tens of billions of dollars you need to compete in search effectively (and that's before you get to competing with their brand). Microsoft has spent over a hundred billion dollars trying to compete with Google, and there's little evidence anybody else has done better anywhere (Western Europe hasn't done anything in search, there's Baidu out of China, and Yandex out of Russia).

VRAM isn't moving nearly as fast as the models are progressing in size. And it's never going to. The cost will get ever greater to operate these at scale.

Unless someone sees a huge paradigm change for cheaper, consumer accessible GPUs in the near future (Intel? AMD? China?). As it is, Nvidia owns the market and they're part of the moat cost problem.

Re: Meta AI releases Code Llama 70B

#140

Post by hackerlight »

turnsout wrote: post #28
jasonjmcghee wrote: post #4

Earlier quoted context omitted.

My personal experience is that Deepseek far exceeds code llama of the same size, but it was released quite a while ago.

Agreed—I hope Meta studied Deepseek's approach. The idea of a Deepseek Coder at 70B would be exciting.

There's a deepseek coder around 30-35b and it has almost identical performance to the 7b on benchmarks.
Post reply on HN