Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI releases Code Llama 70B

twitter.com

Post reply on HN
Print view View original

181–190 of 311 posts

Re: Meta AI releases Code Llama 70B

#181

Post by [deleted] »

coder543 wrote: post #178
Eisenstein wrote: post #177

Earlier quoted context omitted.

The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM. It will be slower, but waiting a couple minutes for a prompt response isn't the worst thing if you are the type that would get use out of a local 70b parameter model. Then again, one could have grabbed 2x 3090s for the price of a 4090 and ended up with 48gb of VRAM in exchange for a very tolerable performance hit.

> The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM. No… that’s not such a great thing. Helpful in a pinch, but if you’re not running at least 70% of your layers on the GPU, then you barely get any benefit from the GPU in my experience. The vast gulf in performance between the CPU and GPU means that the GPU is just spinning its wheels waiting on the CPU. Running half of a model…

[deleted]

Re: Meta AI releases Code Llama 70B

#182

Post by alfalfasprout »

adventured wrote: post #139
svara wrote: post #61

Earlier quoted context omitted.

It's aimed at OpenAI's moat. Making sure they don't accumulate too much of one. No one actually has to use this, it just needs to be clear that LLM as a service won't be super high margin because competition can simply start building on Meta's open source releases.

The moat is all but guaranteed to be the scale of the GPUs required to operate these for a lot of users as they get ever larger, specifically the extreme cost that is going along with that. Anybody have $10 billion sitting around to deploy that gigantic open source set-up for millions of users? There's your moat and only a relatively few companies will be able to do it. One of Google's moats is, has been, and will al…

and this is why the LLM arms race for ultra high parameter count models will stagnate. It's all well and good that we're developing interesting new models. But once you factor cost into the equation it does severely limit what applications justify the cost.

Raw FLOPs may increase each generation but VRAM becomes a limiting factor. And fast VRAM is expensive.

I do expect to see incremental innovation in reducing the size of foundational models.

Re: Meta AI releases Code Llama 70B

#184

Post by SparkyMcUnicorn »

LVB wrote: post #163

I'm not very plugged into how to use these models, but I do love and pay for both ChatGPT and GitHub Copilot. How does one take a model like this (or a smaller version) and leverage it in VS Code? There's a dizzying array of GPT wrapper extensions for VS Code, many of which either seem like kind of junk (10 d/ls, no updates in a year), or just lead to another paid plan, at which point I might as well just keep my GH…

There are some projects that let you run a self-hosted Copilot server, then you set a proxy for the official Copilot extension.

https://github.com/fauxpilot/fauxpilot

https://github.com/danielgross/localpilot

Re: Meta AI releases Code Llama 70B

#185

Post by zaat »

Filligree wrote: post #52
michaelt wrote: post #44

Earlier quoted context omitted.

Why? What benefit is there to grabbing market share from your competitors... in a business you don't even want to be in? By that logic you could justify any bizarre business decision. Should Google launch a social network, to hurt their competitor Facebook? Should Facebook, Amazon and Microsoft each launch a phone?

> Should Google launch a social network, to hurt their competitor Facebook? I mean, Google did launch a social network, to hurt their competitor Facebook. It was a whole thing. It was even a really nice system, eventually.

I enjoyed using Google plus more than any other social network, and managed to create new connections and/or have standard, authentic, real conversations with people I didn't know, most of them ordinary people with shared interests that I would probably wouldn't meet otherwise, some of them are people I can't believe I could connect with directly in any other way - newspapers and news sites editors, major SDK developers. And even with Kevin Kelly.

Re: Meta AI releases Code Llama 70B

#187

Post by hackerlight »

sfsylvester wrote: post #156
theLiminator wrote: post #142

Curious what's the current SOTA local copilot model? Are there any extensions in vscode that give you a similar experience? I'd love something more powerful than copilot for local use (I have a 4090, so I should be able to run a decent number of models).

This is a completely fair, but open question. Not to be a typical HN user, but when you say SOTA local, the question is really what benchmarks do you really care about in order to evaluate. Size, operability, complexity, explainability etc. Working out what copilot models perform best has been a deep exercise for myself and has really made me evaluate my own coding style on what I find important and things I look out…

> https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul... - Also comprehensive, but primarily assesses Python and JavaScript.

I wonder why they didn't use DeepSeek under the "senior" interview test. I am curious to see how it stacks up there.

Re: Meta AI releases Code Llama 70B

#189

Post by sorenjan »

coder543 wrote: post #127
sbrother wrote: post #109

Earlier quoted context omitted.

Can I ask how many tok/s you're getting on that setup? I'm trying to decide whether to invest in a high-end NVIDIA setup or a Mac Studio with llama.cpp for the purposes of running LLMs like this one locally.

On a 33B model at q4_0 quantization, I’m seeing about 36 tokens/s on the RTX 3090 with all layers offloaded to the GPU. Mixtral runs at about 43 tokens/s at q3_K_S with all layers offloaded. I normally avoid going below 4-bit quantization, but Mixtral doesn’t seem phased. I’m not sure if the MoE just makes it more resilient to quantization, or what the deal is. If I run it at q4_0, then it runs at about 24 tokens/s,…

What's the power consumption and fan noise like when doing that? I assume you're running the model doing inference in the background for the whole coding session, i.e. hours at a time?

Re: Meta AI releases Code Llama 70B

#190

Post by sangnoir »

observationist wrote: post #97
eurekin wrote: post #75

Earlier quoted context omitted.

Total speculation: Yann LeCun is there and he is really passionate about the technology and openness

The faux-open models mean the models can't be used in competing products. The open code base means enthusiasts and amateurs and other people hack on Meta projects and contribute improvements. They get free R&D and suppress competition, while looking like they have principles. Yann is clueless about open source principles, or the models would have been Apache or some other comparably open license. It's all ruthless co…

> The faux-open models mean the models can't be used in competing products.

Just because certain entities can't profitably use a product or obtain a license doesn't make it not-open. AGPL is open, for an extreme example.

This argument is also subjective, and not new - "Which is more open BSD-style licenses or GPL?" has ben a guaranteed flameware starter for decades.

Post reply on HN