This will run on a PC with 16GB of ram (possible less) It's a 7b model quantized.
docker run -it --rm ghcr.io/purton-tech/mpt-7b-chat
Ask HN: What are the capabilities of consumer grade hardware to work with LLMs?
11–14 of 14 posts
Re: Ask HN: What are the capabilities of consumer grade hardware to work with LLMs?
#12The short practical answer for inference would be: - You go to https://huggingface.co/TheBloke - You select the model you are interested in - You check which quantizations would fit your available resources.
But what if you are in the market to buy some hardware? Having some rules of thumb won't hurt.
Re: Ask HN: What are the capabilities of consumer grade hardware to work with LLMs?
#13The short practical answer for inference would be: - You go to https://huggingface.co/TheBloke - You select the model you are interested in - You check which quantizations would fit your available resources.
But what if you are in the market to buy some hardware? Having some rules of thumb won't hurt.
Personally I'd go as high as I could afford in this order:
4090 > 3090 > 4080 > 4070 TI > 4070 > 3080 TI > 3080
I have no idea how the 4060 TI 16GB performs
Models change from week to week, so just aim for the basics (VRAM, Tensor cores (generation!), and memory bandwidth).
Re: Ask HN: What are the capabilities of consumer grade hardware to work with LLMs?
#14[flagged]