Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI releases Code Llama 70B

twitter.com

Post reply on HN
Print view View original

301–310 of 311 posts

Re: Meta AI releases Code Llama 70B

#301

Post by annjose »

kungfupawnda wrote: post #300
annjose wrote: post #295

Earlier quoted context omitted.

Yes, it runs a quantized [1] version of the model locally. This version uses low-precision data types to represent reduced weights and activations (8-bit integer instead of 32-bit). The specific model published by Ollama uses 4-bit quantization [2] and that's why it is able to run on MacBook pro. If you want to try it out, this blog post[3] shows how to do it step by step - pretty straightforward. [1] https://hugging…

Thanks, I got all but the 70b model to work. It slows to a crawl on the Mac with 36 gb ram.

Cool. I am running this on M2 Max 64GB. Here is how it looks on my terminal [1]. Btw, the very first run after downloading the model is slightly slow, but the subsequent runs are ok.

[1] https://asciinema.org/a/fFbOEfeTxRShBGbqslwQMfJS4 Note: This recording is in real-time speed, not sped-up.

Re: Meta AI releases Code Llama 70B

#302

Post by [deleted] »

kungfupawnda wrote: post #221
dimask wrote: post #217

Earlier quoted context omitted.

Did you quantise it? At what level and what was your impression compared to other recent smaller models at that quantisation, if so?

No I just ran it out of the box but I had to modify the source code to run for Mac. Instructions here: https://github.com/facebookresearch/llama/pull/947/

[deleted]

Re: Meta AI releases Code Llama 70B

#303

Post by kungfupawnda »

annjose wrote: post #301
kungfupawnda wrote: post #300

Earlier quoted context omitted.

Thanks, I got all but the 70b model to work. It slows to a crawl on the Mac with 36 gb ram.

Cool. I am running this on M2 Max 64GB. Here is how it looks on my terminal [1]. Btw, the very first run after downloading the model is slightly slow, but the subsequent runs are ok. [1] https://asciinema.org/a/fFbOEfeTxRShBGbqslwQMfJS4 Note: This recording is in real-time speed, not sped-up.

I am returning the M3 max 36 GB and picking this model instead. Saves me a grand and it seems to be much more powerful..

Re: Meta AI releases Code Llama 70B

#304

Post by israrkhan »

DaPowaa wrote: post #297
annjose wrote: post #265

Earlier quoted context omitted.

Ollama has released the quantized version. https://ollama.ai/library/codellama:70b https://x.com/ollama/status/1752034686615048367?s=20 Just need to run `ollama run codellama:70b` - pretty fast on macbook.

Really? what kinda macbook pro do you need to run it fast? will a M1 with 16GB ram work? or do we need something super beefy like a M2 fully decked out at 96GB ram to make it run?

M1 with 16GB Ram will barely run codellama:13b

Re: Meta AI releases Code Llama 70B

#305

Post by israrkhan »

theLiminator wrote: post #286
annjose wrote: post #265

Earlier quoted context omitted.

Ollama has released the quantized version. https://ollama.ai/library/codellama:70b https://x.com/ollama/status/1752034686615048367?s=20 Just need to run `ollama run codellama:70b` - pretty fast on macbook.

Do you know how much vram is required?

if you are asking about Apple silicon based Macs, they have integerated GPUs and do not have dedicated graphics memory (UMA)

For running 4 bit quantized model, with 70B parameters you will need around 35G Ram to load it in the memory. So I sould say a Mac with at least 48G memory. That is M3 Max.

Re: Meta AI releases Code Llama 70B

#307

Post by just_testing »

ado__dev wrote: post #238
LVB wrote: post #163

I'm not very plugged into how to use these models, but I do love and pay for both ChatGPT and GitHub Copilot. How does one take a model like this (or a smaller version) and leverage it in VS Code? There's a dizzying array of GPT wrapper extensions for VS Code, many of which either seem like kind of junk (10 d/ls, no updates in a year), or just lead to another paid plan, at which point I might as well just keep my GH…

You can try it with Sourcegraph Cody. https://sourcegraph.com/cody And instructions on how to change the provider to use Ollama w/ whatever model you want: Install and run Ollama - Put ollama in your $PATH. E.g. ln -s ./ollama /usr/local/bin/ollama. - Download Code Llama 70b: ollama pull codellama:70b - Update Cody's VS Code settings to use the unstable-ollama autocomplete provider. - Confirm Cody uses Ollama by look…

Hey, thanks for the tip.

One issue, though: I took a look at the Cody website and it looks like one can't have unlimited completions even when self-hosting a LLM.

I understand you guys have a business model and need to make money out of it. I'm just asking because I work as a teacher and I have students who can't pay an extra subscription and/or students who want to hack into stuff.

Re: Meta AI releases Code Llama 70B

#308

Post by just_testing »

sestinj wrote: post #224
speedgoose wrote: post #179

Earlier quoted context omitted.

Continue doesn’t support tab completion like Copilot yet. A pull/merge request is being worked on: https://github.com/continuedev/continue/pull/758

Release coming later this week!

Thanks! I'm going to recommend it for my students who are interested in tinkering with LLMs.

Re: Meta AI releases Code Llama 70B

#309

Post by mkevac »

kungfupawnda wrote: post #303
annjose wrote: post #301

Earlier quoted context omitted.

Cool. I am running this on M2 Max 64GB. Here is how it looks on my terminal [1]. Btw, the very first run after downloading the model is slightly slow, but the subsequent runs are ok. [1] https://asciinema.org/a/fFbOEfeTxRShBGbqslwQMfJS4 Note: This recording is in real-time speed, not sped-up.

I am returning the M3 max 36 GB and picking this model instead. Saves me a grand and it seems to be much more powerful..

What do you mean you are returning it? It has been used already.

Re: Meta AI releases Code Llama 70B

#310

Post by reddit_clone »

MeImCounting wrote: post #247
int_19h wrote: post #227

Earlier quoted context omitted.

A single RTX 4090 can run at most 34b models with 4-bit quantization. You'd need 2-bit for 70b, and at that point quality plummets. Compute is actually not that big of a deal once generation is ongoing, compared to memory bandwidth. But the initial prompt processing can easily be an order of magnitude slower on CPU, so for large prompts (which would be the case for code completion), acceleration is necessary.

Thats a good point. For example both the RTX 4090 and the RTX 6000 Ada Generation use the AD102 chip. The RTX 6000 Ada though, would be able to run 70b models due to the larger memory pool despite having the same memory interface width.

Would the not-yet-released M3 Mac Mini with upgraded RAM be enough to do some beginner level LLM work?
Post reply on HN