11–20 of 41 posts
Per this calculator, for training, only gpt2-large and gpt2-medium would work with those two top-of-the-line GPUs.
For inference it's certainly a bit better, only the Llama-2-70b-hf and Llama-2-13b-hf don't fit in that much VRAM, all the other models do.
Consumer grade GPUs like NVidia's 3090 and 4090 max out at 24 GB VRAM, and those cost $1000-2000 each. You can get higher VRAM but need enterprise GPUs which are in the five figures, easily starting at $30K a pop. Per this calculator, for training, only gpt2-large and gpt2-medium would work with those two top-of-the-line GPUs. For inference it's certainly a bit better, only the Llama-2-70b-hf and Llama-2-13b-hf don't…
Very large models have to be distributed across multiple GPUs though, even if you’re using datacenter chips like H100s.
Are people still rawdoggin' 16-bit models? I almost exclusively use 5-bit inference quants (or 8-bit natives like Yi-34b) on my MacBook Pro. Tiny accuracy loss, runs fast, and leave plenty of (V)RAM on the table. Mixtral 8x7 is my new daily driver, and only takes like 40GB to run! I wonder if I could run two of them talking to each other...
Are people still rawdoggin' 16-bit models? I almost exclusively use 5-bit inference quants (or 8-bit natives like Yi-34b) on my MacBook Pro. Tiny accuracy loss, runs fast, and leave plenty of (V)RAM on the table. Mixtral 8x7 is my new daily driver, and only takes like 40GB to run! I wonder if I could run two of them talking to each other...
Pure 16bit is horrible for training, sorry.
Consumer grade GPUs like NVidia's 3090 and 4090 max out at 24 GB VRAM, and those cost $1000-2000 each. You can get higher VRAM but need enterprise GPUs which are in the five figures, easily starting at $30K a pop. Per this calculator, for training, only gpt2-large and gpt2-medium would work with those two top-of-the-line GPUs. For inference it's certainly a bit better, only the Llama-2-70b-hf and Llama-2-13b-hf don't…
Nvidia’s workstation cards are available with more RAM than the consumer cards, at a lower price than the datacenter cards. RTX 6000 Ada has 48 GB VRAM and retails for $6800, and RTX 5000 Ada has 32 GB VRAM and retails for $4000[1]. Very large models have to be distributed across multiple GPUs though, even if you’re using datacenter chips like H100s. [1] https://store.nvidia.com/en-us/nvidia-rtx/store/
A single $6800 RTX 6000 Ada with 48GB of VRAM vs 6x 7900XTX with a combined total of 144GB of VRAM honestly makes this seem like a no brainer to me.
This only lists first moments, but Adam stores estimates of first and second moments.
Does this have an option for quantization levels? Don't think I saw it.
Earlier quoted context omitted.
Pure 16bit is horrible for training, sorry.
Doesn't using bf16 alleviate the problem? At least I've had success training a Bert like model from scratch