Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI releases Code Llama 70B

twitter.com

Post reply on HN
Print view View original

241–250 of 311 posts

Re: Meta AI releases Code Llama 70B

#241

Post by Eisenstein »

int_19h wrote: post #222
Eisenstein wrote: post #177

Earlier quoted context omitted.

The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM. It will be slower, but waiting a couple minutes for a prompt response isn't the worst thing if you are the type that would get use out of a local 70b parameter model. Then again, one could have grabbed 2x 3090s for the price of a 4090 and ended up with 48gb of VRAM in exchange for a very tolerable performance hit.

GGUF is just a file format. The ability to offload some layers to CPU is not specific to it nor to llama.cpp in general - indeed, it was available before llama.cpp was even a thing.

I'm pretty sure I didn't assert that it was more that a file format or that llama.cpp was a pioneer in that regard?

Re: Meta AI releases Code Llama 70B

#242

Post by ignoramous »

apapapa wrote: post #234
LVB wrote: post #163

I'm not very plugged into how to use these models, but I do love and pay for both ChatGPT and GitHub Copilot. How does one take a model like this (or a smaller version) and leverage it in VS Code? There's a dizzying array of GPT wrapper extensions for VS Code, many of which either seem like kind of junk (10 d/ls, no updates in a year), or just lead to another paid plan, at which point I might as well just keep my GH…

Free Bard is better than free ChatGPT... Not sure about paid versions

Bard censorship is annoying. One thing I've found (free) Bard to better than the rest at is summarizing book chapters, manuals, and docs. It is also surprisingly good at translation (X to English), as it often adds context to what its translating.

With careful prompt engineering, you can get a lot out of free Bard except when its censored.

Re: Meta AI releases Code Llama 70B

#243

Post by emporas »

yodsanklai wrote: post #210
emporas wrote: post #128

Earlier quoted context omitted.

Yan Le Cunn has talked about Meta's strategy with open source. The general idea, is that the smartest people in the world do not work for you. No company can replicate innovation from open source internally.

> The general idea, is that the smartest people in the world do not work for you Most likely, they work for your competitors. They may not be working to improve your system for free. > No company can replicate innovation from open source internally. Lot of innovation does come from companies.

>Lot of innovation does come from companies.

Of course, i am not arguing that. But when it comes to software as general as code generation, or text generation, the possible applications are so broad, that a team of A.I. researchers in a company, however talented and productive they are, cannot possibly optimize it for every possible use case.

That's what Yan Le Cunn is referring to, and i agree with him. There are a lot of companies which push deep learning forward, and do not release their code or weights freely.

Re: Meta AI releases Code Llama 70B

#245

Post by bredren »

siilats wrote: post #235

We made a Jetbrains plugin called CodeGPT to run this locally https://plugins.jetbrains.com/plugin/21056-codegpt

Are seamless conversations still handled, using the truncation method described in #68?

I was curious if some kind of summary or compression of old exchanges flagged as such might allow the app to remember stuff that had been discussed but fallen outside the token limit.

But possibly request key details lost during summary to bring them back into the new context.

I had thought chatgpt was doing something like this but haven’t read about it.

Re: Meta AI releases Code Llama 70B

#246

Post by observationist »

sangnoir wrote: post #190
observationist wrote: post #97

Earlier quoted context omitted.

The faux-open models mean the models can't be used in competing products. The open code base means enthusiasts and amateurs and other people hack on Meta projects and contribute improvements. They get free R&D and suppress competition, while looking like they have principles. Yann is clueless about open source principles, or the models would have been Apache or some other comparably open license. It's all ruthless co…

> The faux-open models mean the models can't be used in competing products. Just because certain entities can't profitably use a product or obtain a license doesn't make it not-open. AGPL is open, for an extreme example. This argument is also subjective, and not new - "Which is more open BSD-style licenses or GPL?" has ben a guaranteed flameware starter for decades.

I'm not arguing about BSD or GPL. I'm saying that the "open source code, proprietary binary blob" pattern Meta is running with is about quashing potential competition, market positioning, and corporate priorities over any tangential beneficial contributions to open source AI.

It's shitty when other companies do it. It's shitty when Broadcom does it. It's shitty when Meta does it.

It's never a not shitty thing to do.

Re: Meta AI releases Code Llama 70B

#247

Post by MeImCounting »

int_19h wrote: post #227
MeImCounting wrote: post #134

Earlier quoted context omitted.

It pretty much comes down to 2 factors which is memory bandwidth and compute. You need a high enough memory bandwidth to be able to "feed" the compute and you need beefy enough compute to be able to keep up with the data that is being fed in by the memory. In theory a single Nvidia 4090 would be able to run a 70b model with quantization at "useable" speeds. The reason mac hardware is so capable in AI is because of th…

A single RTX 4090 can run at most 34b models with 4-bit quantization. You'd need 2-bit for 70b, and at that point quality plummets. Compute is actually not that big of a deal once generation is ongoing, compared to memory bandwidth. But the initial prompt processing can easily be an order of magnitude slower on CPU, so for large prompts (which would be the case for code completion), acceleration is necessary.

Thats a good point.

For example both the RTX 4090 and the RTX 6000 Ada Generation use the AD102 chip. The RTX 6000 Ada though, would be able to run 70b models due to the larger memory pool despite having the same memory interface width.

Re: Meta AI releases Code Llama 70B

#248

Post by apapapa »

ignoramous wrote: post #242
apapapa wrote: post #234

Earlier quoted context omitted.

Free Bard is better than free ChatGPT... Not sure about paid versions

Bard censorship is annoying. One thing I've found (free) Bard to better than the rest at is summarizing book chapters, manuals, and docs. It is also surprisingly good at translation (X to English), as it often adds context to what its translating. With careful prompt engineering, you can get a lot out of free Bard except when its censored.

Not sure why you talk about Bard's censorship and not Chatgpt's but cause in my experiences it is much worst.

Re: Meta AI releases Code Llama 70B

#249

Post by tinco »

reddit_clone wrote: post #121
moyix wrote: post #59

Earlier quoted context omitted.

You can run it on a Macbook M1/M2 with 64GB of RAM.

I am not too familiar with LLMs and GPUs (Not a gamer either). But want to learn. Could you please expand on what else would be capable of running such models locally? How about a linux laptop/desktop with specific hardware configuration?

There are no other laptops on the market with as much VRAM as the 64GB MBP's have as far as I know. You could make a Linux desktop computer with two 3090's, linked together giving 48GB of VRAM. Wich apparently can run a 4-bit quantized 6k context 70B llama model.

People are recommending Macbooks because they're a relatively cheap and easy way to get a very large amount of RAM hooked up to your accelerator.

Note that these are quantized versions of the model, so they're not as good as the original 70B model, though people claim their performance is really close to original performance. To run without quantization you'd need about 140GB of VRAM. Which would only be possible with an NVidia H100 (don't know the price) or two A100's (at $18,000 each).

Re: Meta AI releases Code Llama 70B

#250

Post by two_in_one »

Philpax wrote: post #85
bk146 wrote: post #70

Can someone explain Meta's strategy with the open source models here? Genuine question, I don't fully undestand. (Please don't say "commoditize your complement" without explaining what exactly they're commoditizing...)

Aside from the "positive" explanations offered in the sibling comments, there's also a "negative" one: other AI companies that try to enter the fray will not be able to compete with Meta's open offerings. After all, why would you pay a company to undertake R&D on building their own models when you can just finetune a Llama?

Whatever Meta's motivation is they help diversify models suppliers. Which is a good thing not to be locked in. As usual reality is more complicated with many moving part. Free models may undercut small startups. But at the same time they stimulate secondary market of providers and tuners.
Post reply on HN