Show HN: I made a GPU VRAM calculator for transformer-based models

#1

Post by furiousteabag » Tue, Dec 26, 2023, 5:53 PM UTC

Show HN: I made a GPU VRAM calculator for transformer-based models
vram.asmirnov.xyz

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#2

Post by cchance » Tue, Dec 26, 2023, 8:27 PM UTC

Very nice, would be cool to have a little i next to each spot to explain what each thing is for newer users (batch size, etc)

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#3

Post by a2128 » Tue, Dec 26, 2023, 8:52 PM UTC

I noticed the default parameter count value is 1.418 billion but if you erase it you can't actually enter it back because you can't type a decimal point in the input area. Also, you can't enter parameter counts smaller than 1 billion

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#4

Post by ilaksh » Tue, Dec 26, 2023, 9:53 PM UTC

Does this have an option for quantization levels? Don't think I saw it.

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#5

Post by sp332 » Tue, Dec 26, 2023, 10:01 PM UTC

a2128 wrote: post #3

I noticed the default parameter count value is 1.418 billion but if you erase it you can't actually enter it back because you can't type a decimal point in the input area. Also, you can't enter parameter counts smaller than 1 billion

It works if you type the digits first and then insert the decimal point after.

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#6

Post by roseway4 » Tue, Dec 26, 2023, 10:13 PM UTC

While not as pretty (and mobile-friendly) as the original link, the calculators below support modeling LoRA-based training, alongside full finetuning.

https://huggingface.co/spaces/Vokturz/can-it-run-llm

https://rahulschand.github.io/gpu_poor/

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#7

Post by ComputerGuru » Tue, Dec 26, 2023, 10:22 PM UTC

ilaksh wrote: post #4

Does this have an option for quantization levels? Don't think I saw it.

I second the request for quantization, eg for exl2.

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#8

Post by ComputerGuru » Tue, Dec 26, 2023, 10:29 PM UTC

roseway4 wrote: post #6

While not as pretty (and mobile-friendly) as the original link, the calculators below support modeling LoRA-based training, alongside full finetuning. https://huggingface.co/spaces/Vokturz/can-it-run-llm https://rahulschand.github.io/gpu_poor/

They seem to be broken when I try any HF ids besides what came preconfigured. e.g. just tried brucethemoose/Yi-34B-200K-DARE-merge-v5-3.1bpw-exl2-fiction or LoneStriker/shisa-7b-v1-3.0bpw-h6-exl2

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#9

Post by a_wild_dandan » Tue, Dec 26, 2023, 10:45 PM UTC

Are people still rawdoggin' 16-bit models? I almost exclusively use 5-bit inference quants (or 8-bit natives like Yi-34b) on my MacBook Pro. Tiny accuracy loss, runs fast, and leave plenty of (V)RAM on the table. Mixtral 8x7 is my new daily driver, and only takes like 40GB to run! I wonder if I could run two of them talking to each other...

Re: Show HN: I made a GPU VRAM calculator for transformer-based models

#10

Post by icelancer » Tue, Dec 26, 2023, 10:56 PM UTC

roseway4 wrote: post #6

While not as pretty (and mobile-friendly) as the original link, the calculators below support modeling LoRA-based training, alongside full finetuning. https://huggingface.co/spaces/Vokturz/can-it-run-llm https://rahulschand.github.io/gpu_poor/

Second link hasn't been working for awhile.