Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta AI releases Code Llama 70B

twitter.com

Post reply on HN
Print view View original

201–210 of 311 posts

Re: Meta AI releases Code Llama 70B

#201

Post by israrkhan »

SparkyMcUnicorn wrote: post #184
LVB wrote: post #163

I'm not very plugged into how to use these models, but I do love and pay for both ChatGPT and GitHub Copilot. How does one take a model like this (or a smaller version) and leverage it in VS Code? There's a dizzying array of GPT wrapper extensions for VS Code, many of which either seem like kind of junk (10 d/ls, no updates in a year), or just lead to another paid plan, at which point I might as well just keep my GH…

There are some projects that let you run a self-hosted Copilot server, then you set a proxy for the official Copilot extension. https://github.com/fauxpilot/fauxpilot https://github.com/danielgross/localpilot

I tried fauxpilot to make it work on my own llama.cpp instance, but didn't work out of the box. Filed a github issue, but did not get any traction. Eventually gave up on it. This was around 5 months ago. Things might have improved by now.

Re: Meta AI releases Code Llama 70B

#202

Post by israrkhan »

petercooper wrote: post #168
LVB wrote: post #163

I'm not very plugged into how to use these models, but I do love and pay for both ChatGPT and GitHub Copilot. How does one take a model like this (or a smaller version) and leverage it in VS Code? There's a dizzying array of GPT wrapper extensions for VS Code, many of which either seem like kind of junk (10 d/ls, no updates in a year), or just lead to another paid plan, at which point I might as well just keep my GH…

https://continue.dev/ is a good place to start.

This looks really good..

Re: Meta AI releases Code Llama 70B

#204

Post by coder543 »

jameshart wrote: post #199
coder543 wrote: post #164

Earlier quoted context omitted.

The impact on accuracy is somewhere in the single-digit percentages at 4-bit quantization, from what I’ve been able to gather. Very small impact. To draw the analogy out further, if the model was able to get an A on a test before quantization, it would likely still get a B at worst afterwards, given a drop in the score of less than 10%. Depending on the task, the measured impact could even be negligible. It’s far mor…

But we’re talking about a coding LLM here. A single digit percentage reduction in accuracy means, what, one or two times in a hundred, it writes == instead of !=?

I think that’s too simplified. The best LLMs will still frequently make mistakes. Meta is advertising a HumanEval score of 67.8%. In a third of cases, the code generated still doesn’t satisfactorily solve the problem in that automated benchmark. The additional errors that quantization would introduce would only be a very small percentage of the overall errors, making the quantized and unquantized models practically indistinguishable to a human observer. Beyond that, lower accuracy can manifest in many ways, and “do the opposite” seems unlikely to be the most common way. There might be a dozen correct ways to solve a problem. The quantized model might choose a different path that still turns out to work, it’s just not exactly the same path.

As someone else pointed out, FLAC is objectively more accurate than mp3, but how many people can really tell? Is it worth 3x the data to store/stream music in FLAC?

The quantized model would run at probably 4x the speed of the unquantized model, assuming you had enough memory to choose between them. Is speed worth nothing? If I have to wait all day for the LLM to respond, I can probably do the work faster myself without its help. Is being able to fit the model onto the hardware you have worth nothing?

In essence, quantization here is a 95% “accurate” implementation of a 67% accurate model, which yields a 300% increase in speed while using just 25% of the RAM. All numbers are approximate, even the HumanEval benchmark should be taken with a large grain of salt.

If you have a very opulent computational experience, you can enjoy the luxury of the full 67.8% accurate model, but that just feels both wasteful and like a bad user experience.

Re: Meta AI releases Code Llama 70B

#205

Post by water-data-dude »

SparkyMcUnicorn wrote: post #184
LVB wrote: post #163

I'm not very plugged into how to use these models, but I do love and pay for both ChatGPT and GitHub Copilot. How does one take a model like this (or a smaller version) and leverage it in VS Code? There's a dizzying array of GPT wrapper extensions for VS Code, many of which either seem like kind of junk (10 d/ls, no updates in a year), or just lead to another paid plan, at which point I might as well just keep my GH…

There are some projects that let you run a self-hosted Copilot server, then you set a proxy for the official Copilot extension. https://github.com/fauxpilot/fauxpilot https://github.com/danielgross/localpilot

When I was setting up a local LLM to play with I stood up my own Open AI API compatible server using llama-cpp-python. I installed the Copilot extension and set OverrideProxyUrl in the advanced settings to point to my local server, but CoPilot obstinately refused to let me do anything until I’d signed in to GitHub to prove that I had a subscription.

I don’t _believe_ that either of these lets you bypass that restriction (although I’d love to be proven wrong), so if you don’t want to sign up for a subscription you’ll need to use something like Continue.

Re: Meta AI releases Code Llama 70B

#207

Post by yodsanklai »

siquick wrote: post #200
crowcroft wrote: post #123

Earlier quoted context omitted.

Meta's end goal is to have better AI than everyone else, in the medium term that means they want to have the best foundational models. How does this help. 1. They become an attractive place for AI researchers to work, and can bring in better staff. 2. They make it less appealing for startups to enter the space and build large foundation models (Meta would prefer 1,000 startups pop up and play around with other people…

> Essentially this puts Meta, Google, and OpenAI/Microsoft (Anthropic/Amazon as a number four maybe) as the only real players in the cutting edge foundational model space. Mistral is right up there.

Mistral has ~20 employees. I'm sure they have good researchers, but don't they lack the computing and engineering resources the big actors have?

Re: Meta AI releases Code Llama 70B

#209

Post by marinhero »

LVB wrote: post #163

I'm not very plugged into how to use these models, but I do love and pay for both ChatGPT and GitHub Copilot. How does one take a model like this (or a smaller version) and leverage it in VS Code? There's a dizzying array of GPT wrapper extensions for VS Code, many of which either seem like kind of junk (10 d/ls, no updates in a year), or just lead to another paid plan, at which point I might as well just keep my GH…

You can download it and run it with [this](https://github.com/oobabooga/text-generation-webui). There's an API mode that you could leverage from your VS Code extension.

Re: Meta AI releases Code Llama 70B

#210

Post by yodsanklai »

emporas wrote: post #128
bk146 wrote: post #70

Can someone explain Meta's strategy with the open source models here? Genuine question, I don't fully undestand. (Please don't say "commoditize your complement" without explaining what exactly they're commoditizing...)

Yan Le Cunn has talked about Meta's strategy with open source. The general idea, is that the smartest people in the world do not work for you. No company can replicate innovation from open source internally.

> The general idea, is that the smartest people in the world do not work for you

Most likely, they work for your competitors. They may not be working to improve your system for free.

> No company can replicate innovation from open source internally.

Lot of innovation does come from companies.

Post reply on HN