Is there a copilot-like autocomplete vscode plugin that uses Ollama?
Yes, the continue.dev plugin can use ollama as backend
Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B
61–65 of 65 posts
Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B
#62Earlier quoted context omitted.
I think the biggest selling point of ollama (llama.cpp) are quantizations, for a slight hit (with q8 or q4) in quality you can get a significant performance boost.
Does ollama/llama.cpp provide low bit operations (avx or cuda kernels) to speed up inference? Or just model compression with inference still done in fp16? My understanding is the modern quantization algorithms are typically implemented in Pytorch.
Sorry I don't know much about this topic.
The only thing I know (from using it) that with quantization I can fit models like llama2 13b, in my 24GB of VRAM when I use q8 (16GB) instead of fp16 (26GB). This means I can get nearly the full quality of llama2 13b's output while still being able to use only my GPU, without the need to do very slow inference on only CPU+RAM.
And the models are quantized before inference, so I'd only download 16GB for the llama2 13b q8 instead of the full 26GB, which means it's not done on the fly.
Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B
#63Earlier quoted context omitted.
Does ollama/llama.cpp provide low bit operations (avx or cuda kernels) to speed up inference? Or just model compression with inference still done in fp16? My understanding is the modern quantization algorithms are typically implemented in Pytorch.
Sorry I don't know much about this topic. The only thing I know (from using it) that with quantization I can fit models like llama2 13b, in my 24GB of VRAM when I use q8 (16GB) instead of fp16 (26GB). This means I can get nearly the full quality of llama2 13b's output while still being able to use only my GPU, without the need to do very slow inference on only CPU+RAM. And the models are quantized before inference, s…
As an aside, even gpt4 level quality does not feel satisfactory to me lately. I can’t imagine willingly using models as dumb as llama2-13b. What do you do with it?
Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B
#64this looks very awesome. can someone tell me why there is no chatter about this? is there something else out there that blows this out of the water in terms of ease of use and access to sample many LLM's ?
Opposite reaction here. I was just thinking, man I hear about Ollama every single day on HN. Not sure a point release is news :-)
Re: Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B
#65Earlier quoted context omitted.
Sorry I don't know much about this topic. The only thing I know (from using it) that with quantization I can fit models like llama2 13b, in my 24GB of VRAM when I use q8 (16GB) instead of fp16 (26GB). This means I can get nearly the full quality of llama2 13b's output while still being able to use only my GPU, without the need to do very slow inference on only CPU+RAM. And the models are quantized before inference, s…
As an aside, even gpt4 level quality does not feel satisfactory to me lately. I can’t imagine willingly using models as dumb as llama2-13b. What do you do with it?
Yeah I agree, everytime a new model releases I download the highest quantization or fp16, that fits into my VRAM, test it out with a few prompts, and then realize that downloadable models are still not as good as the closed ones (except speed wise).
I don't know why I still do it, but everytime I read so many comments how good model X is, and how it outperforms anything else, and then I want to see it for myself.