Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are the capabilities of consumer grade hardware to work with LLMs?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 14 posts

Ask HN: What are the capabilities of consumer grade hardware to work with LLMs?

#1

Post by HexDecOctBin »

If one wants to play with LLMs locally, it is very difficult to find out what one's existing hardware can do – partly because most existing documentation either uses the maximal amount of cloud compute, or is written by startups hoping to sell their own services.

So, given someone has a decent gaming PC with a CUDA-compatible GPU (only Nvidia, I guess?), what can they do with it when it comes to LLMs? What parameter size models can be loaded for various VRAM sizes – for inference, fine tuning and training respectively?

Let's say the VRAM sizes are 8 GB, 12 GB, 16 GB and 24 GB, which seem to be the most common in the 40x0 series of GPUs. If system RAM matters, what can be done with 16 GB, 32 GB, 64 GB and beyond?

Re: Ask HN: What are the capabilities of consumer grade hardware to work with LLMs?

#3

Post by TheRealSteel »

I agree, I've definitely seen way more information about running image synthesis models like Stable Diffusion locally than I have LLMs. It's counterintuitive to me that Stable Diffusion takes less RAM than an LLM, especially considering it still needs the word vectors. Goes to show I know nothing.

I guess it comes down to the requirement of a very high end (or multiple) GPU that makes it impractical for most vs just running it in Colab or something.

Tho there are some efforts:

https://github.com/cocktailpeanut/dalai

Re: Ask HN: What are the capabilities of consumer grade hardware to work with LLMs?

#4

Post by HexDecOctBin »

TheRealSteel wrote: post #3

I agree, I've definitely seen way more information about running image synthesis models like Stable Diffusion locally than I have LLMs. It's counterintuitive to me that Stable Diffusion takes less RAM than an LLM, especially considering it still needs the word vectors. Goes to show I know nothing. I guess it comes down to the requirement of a very high end (or multiple) GPU that makes it impractical for most vs just…

Even the README of this project is lacking in details. They cover sizes of system RAM, but not of VRAM. Are they running the models on CPU exclusively? How does that compare to running them on GPUs? What are the corresponding numbers for GPUs?

Not to mention more meta-topics, like which is preferred for inferencing - CPU or GPU? What are the corresponding numbers for fine-tuning or training for various model sizes? and so on.

Re: Ask HN: What are the capabilities of consumer grade hardware to work with LLMs?

#5

Post by [deleted] »

HexDecOctBin wrote: post #4
TheRealSteel wrote: post #3

I agree, I've definitely seen way more information about running image synthesis models like Stable Diffusion locally than I have LLMs. It's counterintuitive to me that Stable Diffusion takes less RAM than an LLM, especially considering it still needs the word vectors. Goes to show I know nothing. I guess it comes down to the requirement of a very high end (or multiple) GPU that makes it impractical for most vs just…

Even the README of this project is lacking in details. They cover sizes of system RAM, but not of VRAM. Are they running the models on CPU exclusively? How does that compare to running them on GPUs? What are the corresponding numbers for GPUs? Not to mention more meta-topics, like which is preferred for inferencing - CPU or GPU? What are the corresponding numbers for fine-tuning or training for various model sizes? a…

[deleted]

Re: Ask HN: What are the capabilities of consumer grade hardware to work with LLMs?

#6

Post by sp332 »

GPU will be faster if you can fit the data in VRAM. If not, using CPU and system RAM works fine but is slower. It's even possible to load layers from disk, but this is very very slow.

Full precision for most models is 16 bits. That means two bytes per parameter. This is a rule of thumb and there's other overheard, but generally, you can load a 7B model in ~14GB of VRAM or system RAM at full precision. But usually, to improve speed and memory usage, precision is reduced after training. Loading a model at 8-bit precision means you can fit a 13B model in ~13GB of (V)RAM. You can go even lower, with 4 bits being common and 3 or 2 bits available for the most popular large models.

GPT4all has the least setup friction but also a pretty limited interface. Last I checked though, if you're on Windows, it will not run on GPU. Building and installing llama.cpp from source is quite painless and has more options. Installing text-generation-webui has a lot more options in exchange for a few more steps in the install. Those are the top 3 I would recommend for ease of use.

Edit: all above numbers are for inference. For training numbers, https://huggingface.co/docs/transformers/perf_train_gpu_one is pretty approachable. tl;dr most commonly 8 bytes per parameter, but optimizations are possible with tradeoffs in complexity and accuracy.

Re: Ask HN: What are the capabilities of consumer grade hardware to work with LLMs?

#7

Post by HexDecOctBin »

sp332 wrote: post #6

GPU will be faster if you can fit the data in VRAM. If not, using CPU and system RAM works fine but is slower. It's even possible to load layers from disk, but this is very very slow. Full precision for most models is 16 bits. That means two bytes per parameter. This is a rule of thumb and there's other overheard, but generally, you can load a 7B model in ~14GB of VRAM or system RAM at full precision. But usually, to…

Than you. Just to confirm, this for inferencing?

Are there any similar heuristics for fine-tuning an LLM on personal data?

Re: Ask HN: What are the capabilities of consumer grade hardware to work with LLMs?

#10

Post by james-revisoai »

TheRealSteel wrote: post #3

I agree, I've definitely seen way more information about running image synthesis models like Stable Diffusion locally than I have LLMs. It's counterintuitive to me that Stable Diffusion takes less RAM than an LLM, especially considering it still needs the word vectors. Goes to show I know nothing. I guess it comes down to the requirement of a very high end (or multiple) GPU that makes it impractical for most vs just…

Well, if you looked at the AI world in 2019, the models were mostly less RAM intense (typically 0.8GB-2GB RAM for BERT/T5 non-large for example). And computer vision models always hit the max you'd throw at them.

Things twist and turn. Text requires a large amount of parameters to be useful. If we as humans required 8000x8000 pictures to discern objects, images would require more. It's quite anthropomorphic. That's really the core of it.

Post reply on HN