Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I made a GPU VRAM calculator for transformer-based models

vram.asmirnov.xyz

Post reply on HN
Print view View original

21–30 of 41 posts

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#21

Post by ttt3ts »

slabity wrote: post #16
alexhutcheson wrote: post #12

Earlier quoted context omitted.

Nvidia’s workstation cards are available with more RAM than the consumer cards, at a lower price than the datacenter cards. RTX 6000 Ada has 48 GB VRAM and retails for $6800, and RTX 5000 Ada has 32 GB VRAM and retails for $4000[1]. Very large models have to be distributed across multiple GPUs though, even if you’re using datacenter chips like H100s. [1] https://store.nvidia.com/en-us/nvidia-rtx/store/

Other than power consumption, is there any reason to prefer a single workstation card over multiple consumer cards then? A single $6800 RTX 6000 Ada with 48GB of VRAM vs 6x 7900XTX with a combined total of 144GB of VRAM honestly makes this seem like a no brainer to me.

You have to pass the context between GPUs for large models that don't fit in VRAM. Often ends up slower. Also, tooling around AMD GPUs is still poor in comparison.

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#22

Post by alexhutcheson »

slabity wrote: post #16
alexhutcheson wrote: post #12

Earlier quoted context omitted.

Nvidia’s workstation cards are available with more RAM than the consumer cards, at a lower price than the datacenter cards. RTX 6000 Ada has 48 GB VRAM and retails for $6800, and RTX 5000 Ada has 32 GB VRAM and retails for $4000[1]. Very large models have to be distributed across multiple GPUs though, even if you’re using datacenter chips like H100s. [1] https://store.nvidia.com/en-us/nvidia-rtx/store/

Other than power consumption, is there any reason to prefer a single workstation card over multiple consumer cards then? A single $6800 RTX 6000 Ada with 48GB of VRAM vs 6x 7900XTX with a combined total of 144GB of VRAM honestly makes this seem like a no brainer to me.

You can only fit 1-2 graphics cards in a “normal” ATX case (each card takes 2-3 “slots”). If you want 4 cards on one machine, you need a bigger/more expensive motherboard, case, PSU, etc. I haven’t personally seen anyone put 6 cards in a workstation.

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#23

Post by 3abiton »

roseway4 wrote: post #6

While not as pretty (and mobile-friendly) as the original link, the calculators below support modeling LoRA-based training, alongside full finetuning. https://huggingface.co/spaces/Vokturz/can-it-run-llm https://rahulschand.github.io/gpu_poor/

Been looking for something like thos for a while! I googled a lot, and this link never popped up. I feel google search is regressing.

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#24

Post by rdedev »

shikon7 wrote: post #20
rdedev wrote: post #15

Earlier quoted context omitted.

Doesn't using bf16 alleviate the problem? At least I've had success training a Bert like model from scratch

I wonder about that too. With the small precision, parameter updates might be too small to have an effect (is it possible to use some sort of probabilistic update in that case?) Unfortunately, I haven’t found any resources describing the feasibility of full fp16 or bf16 training.

Ah my bad. I am using mixed precision training in the my previous comment.

You might find this paper interesting: https://arxiv.org/pdf/2010.06192.pdf

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#25

Post by bigdict »

rubatuga wrote: post #14
a_wild_dandan wrote: post #9

Are people still rawdoggin' 16-bit models? I almost exclusively use 5-bit inference quants (or 8-bit natives like Yi-34b) on my MacBook Pro. Tiny accuracy loss, runs fast, and leave plenty of (V)RAM on the table. Mixtral 8x7 is my new daily driver, and only takes like 40GB to run! I wonder if I could run two of them talking to each other...

Pure 16bit is horrible for training, sorry.

Hmm, what do you mean? I thought bf16 is used extensively for LLM training.

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#26

Post by furiousteabag »

rdedev wrote: post #15
rubatuga wrote: post #14

Earlier quoted context omitted.

Pure 16bit is horrible for training, sorry.

Doesn't using bf16 alleviate the problem? At least I've had success training a Bert like model from scratch

Mixed precision is a default method to pretrain and full fine tune right now. It is especially good in transformers, because they have memory bottleneck in activations (outputs of intermediate layers stored for backprop), and running forward pass in fp16/bf16 reduces VRAM by almost half (speeds up forward pass as well).

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#28

Post by furiousteabag »

shikon7 wrote: post #20
rdedev wrote: post #15

Earlier quoted context omitted.

Doesn't using bf16 alleviate the problem? At least I've had success training a Bert like model from scratch

I wonder about that too. With the small precision, parameter updates might be too small to have an effect (is it possible to use some sort of probabilistic update in that case?) Unfortunately, I haven’t found any resources describing the feasibility of full fp16 or bf16 training.

You are correct, training sorely in fp16/bf16 can lead to imprecise weight updates or even gradients turning to zero. Because of that, mixed precision is used. In mixed precision training, we keep a copy of the weights in fp32 (master model) and the training loop looks like this: compute the output with the fp16 model, then the loss -> back-propagate the gradients in half-precision -> copy the gradients in fp32 precision -> do the update on the master model (in fp32 precision) -> copy the master model in the fp16 model. We also do loss scaling which means multiplying the output of the loss function by some scalar number before backprop (necessary in fp16 but not required in bf16).

Check out the fastai docs for more details: https://docs.fast.ai/callback.fp16.html

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#29

Post by chrsig »

a_wild_dandan wrote: post #9

Are people still rawdoggin' 16-bit models? I almost exclusively use 5-bit inference quants (or 8-bit natives like Yi-34b) on my MacBook Pro. Tiny accuracy loss, runs fast, and leave plenty of (V)RAM on the table. Mixtral 8x7 is my new daily driver, and only takes like 40GB to run! I wonder if I could run two of them talking to each other...

How does one rawdog a 16-bit model?

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#30

Post by oceanplexian »

alexhutcheson wrote: post #22
slabity wrote: post #16

Earlier quoted context omitted.

Other than power consumption, is there any reason to prefer a single workstation card over multiple consumer cards then? A single $6800 RTX 6000 Ada with 48GB of VRAM vs 6x 7900XTX with a combined total of 144GB of VRAM honestly makes this seem like a no brainer to me.

You can only fit 1-2 graphics cards in a “normal” ATX case (each card takes 2-3 “slots”). If you want 4 cards on one machine, you need a bigger/more expensive motherboard, case, PSU, etc. I haven’t personally seen anyone put 6 cards in a workstation.

In a water cooled config the cards only take 1 slot. I’ve got 2 3090s and am buying another two shortly. Preemtively upgraded the power to 220v, found a 2kw PSU, and installed a dedicated mini split. I’m also undervolting the cards to keep power and heat down, because even 2000w is not enough to run 4 and a server grade CPU without tripping. When you start accumulating GPUs you also run into all kinds of thermal and power problems for the room, too.
Post reply on HN