31–40 of 274 posts
This seems like a great step; I’ve been able to run StableDiffusion locally, but with an older GPU none of the LLMs will run for me since I don’t have enough VRAM. Oddly I don’t see a VRAM requirement listed. Anyone know if it has a lower limit?
I think you can run Pygmalion 6B on a 8GB GPU using DeepSpeed.
It's very underwhelming if you expect something like ChatGPT though.
If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.
This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.
If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.
This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.
Any chance these work on CPUs with any acceptable performance? I have a 10-core 20-thread monster CPU, but didn't bother with a dedicated GPU because I can't control something as simple as its temperature. See the complicated procedure that only works with the large proprietary driver here: https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Over...
Nope. 20 cores in a CPU, 2000 in a GPU, with much much faster memory and an architecture designed to chew through data as fast as possible.
(a difference I think is that there are more virtual lanes, some of may be masked off, that are mapped to the GPU physical SIMD lanes)
This is cool! But I wonder if it's economical using cloud hardware. The author claims 1.12 tokens/second on the 175B parameter model (arguably comparable to GPT-3 Davinci). That's about 100k tokens a day on the GCP machine the author used. Someone double check my numbers here, but given the Davinci base cost of $0.02 per 1k tokens and GCP cost for the hardware listed "NVIIDA T4 (16GB) instance on GCP with 208GB of DR…
Any chance these work on CPUs with any acceptable performance? I have a 10-core 20-thread monster CPU, but didn't bother with a dedicated GPU because I can't control something as simple as its temperature. See the complicated procedure that only works with the large proprietary driver here: https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Over...
To give you an idea of the order of magnitude of compute difference. Sapphire Rapids has AMX and fp16 AVX512 to close the gap a little, but it's still massive.
https://github.com/Ying1123/FlexGen/blob/main/docs/paper.pdf https://docs.google.com/viewer?url=https://github.com/Ying11...
Earlier quoted context omitted.
The OPT175b model is massive. A lot of that system ram probably holds model data.
Interesting, though apparently the OPT175B model is 350GB: > You will need at least 350GB GPU memory on your entire cluster to serve the OPT-175B model. For example, you can use 4 x AWS p3.16xlarge instances, which provide 4 (instance) x 8 (GPU/instance) x 16 (GB/GPU) = 512GB memory. https://alpa.ai/tutorials/opt_serving.html (Scroll down to the second "Note", not far from the top) I wonder what FlexGen is doing.. a…
Any chance these work on CPUs with any acceptable performance? I have a 10-core 20-thread monster CPU, but didn't bother with a dedicated GPU because I can't control something as simple as its temperature. See the complicated procedure that only works with the large proprietary driver here: https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Over...
With what, 50GB/s memory bandwidth? That's no monster. The two consumer GPUs in my machine both do 1TB/s and are still bottlenecked on memory bandwidth.
> only works with the large proprietary driver here
In practice, nothing works without the proprietary driver so this isn't specific to temperature. Also the setting you're looking for is almost certainly `nvidia-smi -pl $watts` for setting the power limit, not whatever that wiki gives you. GPU temperature = ambient temperature + (power limit)*(thermal resistance of cooler)
Earlier quoted context omitted.
I tried to figure out how to do GPGPU stuff as a total beginner in Rust on Apple Silicon. I couldn't figure out if I was supposed to be chasing down Apple Metal or OpenCL backends. It also didn't seem to make much of a difference because while there are crates for both that seemed relatively well-maintained/fleshed out, I couldn't figure out how exactly to just pull one down and plug them into a higher level library…
Give this a look: https://github.com/guillaume-be/rust-bert https://github.com/guillaume-be/rust-bert/blob/master/exampl... If you have Pytorch configured correctly, this should "just work" for a lot of the smaller models. It won't be a 1:1 ChatGPT replacement, but you can build some pretty cool stuff with it. > it's basically Python or bust in this space More or less, but that doesn't have to be a bad thing. If you'…
> Port of Hugging Face's Transformers library, using the tch-rs crate and pre-processing from rust-tokenizers.
> tch-rs: Rust bindings for the C++ api of PyTorch.
Which "backend" does this end up using on Apple Silicon, MPS (Metal Performance Shaders) or OpenCL?
https://pytorch.org/docs/stable/notes/mps.html
I'm going to guess MPS?