221–230 of 311 posts
Earlier quoted context omitted.
I got it to build and run the example app on my M3 max with 36 gb ram. Memory pressure was around 32 gb
Did you quantise it? At what level and what was your impression compared to other recent smaller models at that quantisation, if so?
Instructions here: https://github.com/facebookresearch/llama/pull/947/
Earlier quoted context omitted.
I think your take is a bit optimistic. I like quantization as much as the next person, but even the 2-bit model won’t fit entirely on a 4090: https://huggingface.co/TheBloke/Llama-2-70B-GGUF I would be uncomfortable recommending less than 4-bit quantization on a non-MoE model, which is ~40GB on a 70B model.
The great thing about gguf is that it will cross to system RAM if there isn't enough VRAM. It will be slower, but waiting a couple minutes for a prompt response isn't the worst thing if you are the type that would get use out of a local 70b parameter model. Then again, one could have grabbed 2x 3090s for the price of a 4090 and ended up with 48gb of VRAM in exchange for a very tolerable performance hit.
Not sure who this is aimed at? The avg programmer probably doesn’t have the gear on hand to run this at the required pace Cool nonetheless
Earlier quoted context omitted.
https://continue.dev/ is a good place to start.
This looks really good..
I'm not very plugged into how to use these models, but I do love and pay for both ChatGPT and GitHub Copilot. How does one take a model like this (or a smaller version) and leverage it in VS Code? There's a dizzying array of GPT wrapper extensions for VS Code, many of which either seem like kind of junk (10 d/ls, no updates in a year), or just lead to another paid plan, at which point I might as well just keep my GH…
I’ve been working on continue.dev, which is completely free to use with your own Ollama instance / TogetherAI key, or for a while with ours. Was testing with Codellama-70b this morning and it’s clearly a step up from other OS models
Earlier quoted context omitted.
I am not too familiar with LLMs and GPUs (Not a gamer either). But want to learn. Could you please expand on what else would be capable of running such models locally? How about a linux laptop/desktop with specific hardware configuration?
It pretty much comes down to 2 factors which is memory bandwidth and compute. You need a high enough memory bandwidth to be able to "feed" the compute and you need beefy enough compute to be able to keep up with the data that is being fed in by the memory. In theory a single Nvidia 4090 would be able to run a 70b model with quantization at "useable" speeds. The reason mac hardware is so capable in AI is because of th…
Compute is actually not that big of a deal once generation is ongoing, compared to memory bandwidth. But the initial prompt processing can easily be an order of magnitude slower on CPU, so for large prompts (which would be the case for code completion), acceleration is necessary.
I think Copilot is already highly subsidized by Microsoft.
Let's say you use Copilot around 30% of your daily work hours. How much kWh does an opensource 7B or 13B model use then in a month on one 4090?
EDIT:
I think for a 13B at 30% use per day it comes around 30$/no on energy bill.
So probably with a even more smaller but capable model can beat the Copilot monthly subscription.
Earlier quoted context omitted.
I’ve been working on continue.dev, which is completely free to use with your own Ollama instance / TogetherAI key, or for a while with ours. Was testing with Codellama-70b this morning and it’s clearly a step up from other OS models
How do you test a 70B model locally? I've tried to query, but the response is super slow.
Credit where credit is due, Meta has had a fantastic commitment towards open source ML. You love to see it.