21–30 of 102 posts
I'm all for having more open source projects, but I do not see how it can be useful in this ecosystem, especially for people with newer AMD GPUs (not supported in this project) which are already supported in most popular projects?
For inference, if you have a supported card (or probably architecture if you are on Linux and can use HSA_OVERRIDE_GFX_VERSION), then you can probably run anything with (upstream) PyTorch and transformers. Also, compiling llama.cpp is has been pretty trouble-free for me for at least a year. (If you are on Windows, there is usually a win-hip binary of llama.cpp in the project's releases or if things totally refuse to…
tldr: uses the latest rocm 6.2 to run full precision inference for llama 405b on a single node 8 x MI300x AMD GPU
How mature do you think Rocm 6.2-AMD stack is compared to Nvidia ?
I almost tried to install AMD rocm a while ago after discovering the simplicity of llamafile. sudo apt install rocm Summary: Upgrading: 0, Installing: 203, Removing: 0, Not Upgrading: 0 Download size: 2,369 MB / 2,371 MB Space needed: 35.7 GB / 822 GB available I don't understand how 36 GB can be justified for what amounts to a GPU driver.
It's not just you; AMD manages to completely shit-up the Linux kernel with their drivers: https://www.phoronix.com/news/AMD-5-Million-Lines
For inference, if you have a supported card (or probably architecture if you are on Linux and can use HSA_OVERRIDE_GFX_VERSION), then you can probably run anything with (upstream) PyTorch and transformers. Also, compiling llama.cpp is has been pretty trouble-free for me for at least a year. (If you are on Windows, there is usually a win-hip binary of llama.cpp in the project's releases or if things totally refuse to…
related: https://www.nonbios.ai/post/deploying-large-405b-models-in-f... tldr: uses the latest rocm 6.2 to run full precision inference for llama 405b on a single node 8 x MI300x AMD GPU How mature do you think Rocm 6.2-AMD stack is compared to Nvidia ?
People use "Docker-based" all the time but what they mean is that they ship $SOFTWARE in a Docker image. "Docker-based" reads, to me, as if you were doing Inference on AMD cards with Docker somehow, which doesn't make sense.
https://github.com/slashml/amd_inference/blob/main/run-docke...
Why would you use this over vLLM?
I almost tried to install AMD rocm a while ago after discovering the simplicity of llamafile. sudo apt install rocm Summary: Upgrading: 0, Installing: 203, Removing: 0, Not Upgrading: 0 Download size: 2,369 MB / 2,371 MB Space needed: 35.7 GB / 822 GB available I don't understand how 36 GB can be justified for what amounts to a GPU driver.
It's not just you; AMD manages to completely shit-up the Linux kernel with their drivers: https://www.phoronix.com/news/AMD-5-Million-Lines
What's the best bang-for-your-buck AMD GPU these days? I just bought 2 used 3090s for $750ish refurb'd on eBay. Curious what others are using for running LLMs locally.
Personal experience: It's not even worth it. AMD (i)GPU breaks with every pytorch, ROCm, xformers, or ollama updates. You'll sleep more compfortably at night.