Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI releases Code Llama 70B

twitter.com

Post reply on HN
Print view View original

211–220 of 311 posts

Re: Meta AI releases Code Llama 70B

#211

Post by fngjdflmdflg »

IshKebab wrote: post #111
pchristensen wrote: post #86

Earlier quoted context omitted.

Meta doesn't have an AI "product" competing with OpenAI, Google's Bard, etc. But they use AI extensively internally. This is roughly a byproduct of their internal AI work that they're already doing, and fostering open source AI development puts incredible pressure on the AI products and their owners. If Meta can help prevent there from being an AI monopoly company, but rather an ecosystem of comparable products, then…

Google releasing a (very popular) web browser gives them direct control of web standards. What does this give Facebook?

Web standards are probably the last thing Google cares about with Chrome. Much more important is being the default search engine and making sure data collection isn't interrupted by a potential privacy minded browser.

Re: Meta AI releases Code Llama 70B

#212

Post by idkyall »

api wrote: post #19
dsabanin wrote: post #15

Earlier quoted context omitted.

I think Meta's goal is to subvert Google, MS and OpenAI, after realizing it's not positioned well to compete with them commercially.

Could also be that these smaller models are a loss leader or advertisement for a future product or service... like a big brother to Llama3 that's commercial.

I believe there were rumors they are developing a commercial model: e.g. https://www.ft.com/content/01fd640e-0c6b-4542-b82b-20afb203f...

Re: Meta AI releases Code Llama 70B

#213

Post by esafak »

wanderingstan wrote: post #81
Filligree wrote: post #52

Earlier quoted context omitted.

> Should Google launch a social network, to hurt their competitor Facebook? I mean, Google did launch a social network, to hurt their competitor Facebook. It was a whole thing. It was even a really nice system, eventually.

And it turned out that Facebook had quite a moat with network effects. OpenAI doesn’t have such a moat, which may be what Meta is wanting to expose.

Google botched the launch, and they never nurture products after launch anyway. Google+ could have been more successful.

Re: Meta AI releases Code Llama 70B

#214

Post by dimask »

Eisenstein wrote: post #177
coder543 wrote: post #169

Earlier quoted context omitted.

I think your take is a bit optimistic. I like quantization as much as the next person, but even the 2-bit model won’t fit entirely on a 4090: https://huggingface.co/TheBloke/Llama-2-70B-GGUF I would be uncomfortable recommending less than 4-bit quantization on a non-MoE model, which is ~40GB on a 70B model.

The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM. It will be slower, but waiting a couple minutes for a prompt response isn't the worst thing if you are the type that would get use out of a local 70b parameter model. Then again, one could have grabbed 2x 3090s for the price of a 4090 and ended up with 48gb of VRAM in exchange for a very tolerable performance hit.

> The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM.

Then you can just run it entirely on CPU. There is no point to buy an expensive GPU to run LLMs to be bottlenecked by your CPU in the first place. Which is why I do not get so excited with these huge models, as they gain less traction as not as many people can run them locally, and finetuning is probably more costly too.

Re: Meta AI releases Code Llama 70B

#215

Post by nullc »

eurekin wrote: post #137
kuczmama wrote: post #130

Earlier quoted context omitted.

Wow that's a really good idea. I could potentially buy 4 Nvidia P40's for the same price as a 3090 and run inference on pretty much any model I want.

Just make sure you're comfortable with manually compiling the bitsandbytes and generally combine a software stack of almost out of date libraries

P40 still works with 12.2 at the moment. I used to use K80s (which I think I paid like $50 for!) which turned into a huge mess to deal with older libraries, especially since essentially all ML stuff is on a crazy upgrade cadence with everything constantly breaking even without having to deal with orphaned old software.

You can get gpu server chassis that have 10 pci-slots too! for around $2k on ebay. But note that there is a hardware limitation on the PCI-E cards such that each card can only directly communicate with 8 others at a time. Beware, they're LOUD even by the standards of sever hardware.

Oh also the nvidia tesla power connectors have cpu-connector like polarity instead of pci-e, so at least in my chassis I needed to adapt them.

Also keep in mind that if you aren't using a special gpu chassis, the tesla cards don't have fans, so you have to provide cooling.

Re: Meta AI releases Code Llama 70B

#216

Post by crowcroft »

siquick wrote: post #200
crowcroft wrote: post #123

Earlier quoted context omitted.

Meta's end goal is to have better AI than everyone else, in the medium term that means they want to have the best foundational models. How does this help. 1. They become an attractive place for AI researchers to work, and can bring in better staff. 2. They make it less appealing for startups to enter the space and build large foundation models (Meta would prefer 1,000 startups pop up and play around with other people…

> Essentially this puts Meta, Google, and OpenAI/Microsoft (Anthropic/Amazon as a number four maybe) as the only real players in the cutting edge foundational model space. Mistral is right up there.

I'm curious to see how they go, I might have a limited understanding. From what I can tell they do a good job in terms of value and efficiency with 'lighter' models, but I don't put them in the same category as the others in the sense that they aren't producing the massive absolute best in class LLMs.

Hopefully they can prove me wrong though!

Re: Meta AI releases Code Llama 70B

#217

Post by dimask »

kungfupawnda wrote: post #174
Havoc wrote: post #23

Not sure who this is aimed at? The avg programmer probably doesn’t have the gear on hand to run this at the required pace Cool nonetheless

I got it to build and run the example app on my M3 max with 36 gb ram. Memory pressure was around 32 gb

Did you quantise it? At what level and what was your impression compared to other recent smaller models at that quantisation, if so?

Re: Meta AI releases Code Llama 70B

#218

Post by trentnelson »

kkzz99 wrote: post #42
kuczmama wrote: post #35

Earlier quoted context omitted.

Realistically, what hardware would be required to run this? I assumed a RTX 3090 would be enough?

RTX 3090 has 24GB of memory, a quantized llama70b takes around 60GB of memory. You can offload a few layers on the gpu, but most of them will run on the CPU with terrible speeds.

Can that be split across multiple GPUs? i.e. what if I have 4xV100-DGXS-32GBs?

Re: Meta AI releases Code Llama 70B

#219

Post by sp332 »

coder543 wrote: post #178
Eisenstein wrote: post #177

Earlier quoted context omitted.

The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM. It will be slower, but waiting a couple minutes for a prompt response isn't the worst thing if you are the type that would get use out of a local 70b parameter model. Then again, one could have grabbed 2x 3090s for the price of a 4090 and ended up with 48gb of VRAM in exchange for a very tolerable performance hit.

> The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM. No… that’s not such a great thing. Helpful in a pinch, but if you’re not running at least 70% of your layers on the GPU, then you barely get any benefit from the GPU in my experience. The vast gulf in performance between the CPU and GPU means that the GPU is just spinning its wheels waiting on the CPU. Running half of a model…

The main benefit of a GPU in that case is much faster prompt reading. Could be useful for Code Llama cases where you want the model to read a lot of code and then write a line or part of a line.

Re: Meta AI releases Code Llama 70B

#220

Post by int_19h »

coder543 wrote: post #180
nox101 wrote: post #176

Earlier quoted context omitted.

fortunately it will run on my UMA mac. it's made me curious what the trade offs are. Would I be better off with a 4090 or a Mac with 128+gig of uma memory

Even the M3 Max seems to be slower than my 3090 for LLMs that fit onto the 3090, but it’s hard to find comprehensive numbers. The primary advantage is that you can spec out more memory with the M3 Max to fit larger models, but with the exception of CodeLlama-70B today, it really seems like the trend is for models to be getting smaller and better, not bigger. Mixtral runs circles around Llama2-70B and arguably ChatGPT…

M3 Max is actually less than ideal because it peaks at 400 Gb/s for memory. What you really want is M1 or M2 Ultra, which offers up to 800 Gb/s (for comparison, RTX 3090 runs at 936 GB/s). A Mac Studio suitable for running 70B models with speeds fast enough for realtime chat can be had for ~$3K

The downside of Apple's hardware at the moment is that the training ecosystem is very much focused on CUDA; llama.cpp has an open issue about Metal-accelerated training: https://github.com/ggerganov/llama.cpp/issues/3799 - but no work on it so far. This is likely because training at any significant sizes requires enough juice that it's pretty much always better to do it in the cloud currently, where, again, CUDA is the well-established ecosystem, and it's cheaper and easier for datacenter operators to scale. But, in principle, much faster training on Apple hardware should be possible, and eventually someone will get it done.

Post reply on HN