Ask HN: How to run Language Models on your own?

Probably easier to answer your question if you can explain what you are trying to achieve.

The best of the best right now is probably Vicuna 13B. The 30B and 65B LLaMA models are better on benchmarks, but there isn’t a compelling instruct fine tuned version of those yet so they require a lot of prompt engineering.

If you want to run Vicuna without quantization you need 25GB of VRAM, which exceeds pretty much all consumer GPUs. Vicuna 4bit GPTQ is decent though I personally notice a quality difference when comparing it to 16bit.

CPU is also an option, you can run pretty much any model that will fit in your RAM, although your performance will obviously suffer. LlamaCPP has gotten very popular.

On a slight tangent, I feel like this is one major ace up Apple’s sleeve, if they can zoom into it. With the awesome performance of Apple Silicon and how lots of the big leaps are seen with video rendering, they just need to focus some more on the ML capabilities. They’ve been skirting around ML with some optimizing on popular ML libraries, but it’s mostly focused on inference, but hopefully with recent pytorch 2.0 optimizations and co, they can meet open source libraries halfway, and do more.

I think the current proliferation of AI and general awareness of LLM can be a major selling point if they make sure their neural engine is well optimized for it. Will put them right at the center of the conversation, especially since one of the current concern is the cost of training these models.