Earlier quoted context omitted.
It would be great if you included a section on running with Docker on Linux. The only one that worked out of the box was Ollama, and it had an example. https://github.com/ollama/ollama/blob/main/docs/docker.md has a docker image but no examples to run it https://github.com/ggerganov/llama.cpp/blob/master/docs/dock... has a docker image but no examples to run it https://github.com/LostRuins/koboldcpp?tab=readme-ov-fil…
good feedback thanks, would you be able to open an issue
AMD GPU Inference
61–70 of 102 posts
Re: AMD GPU Inference
#62On Ubuntu 24.04 (and Debian Unstable¹), the OS-provided packages should be able to get llama.cpp running on ROCm on just about any discrete AMD GPU from Vega onwards²³⁴. No docker or HSA_OVERRIDE_GFX_VERSION required. The performance might not be ideal in every case⁵, but I've tested a wide variety of cards:
# install dependencies
sudo apt -y update
sudo apt -y upgrade
sudo apt -y install git wget hipcc libhipblas-dev librocblas-dev cmake build-essential
# ensure you have permissions by adding yourself to the video and render groups
sudo usermod -aG video,render $USER
# log out and then log back in to apply the group changes
# you can run `rocminfo` and look for your GPU in the output to check everything is working thus far
# download a model, build llama.cpp, and run it
wget https://huggingface.co/TheBloke/dolphin-2.2.1-mistral-7B-GGUF/resolve/main/dolphin-2.2.1-mistral-7b.Q5_K_M.gguf?download=true -O dolphin-2.2.1-mistral-7b.Q5_K_M.gguf
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
git checkout b3267
HIPCXX=clang-17 cmake -H. -Bbuild -DGGML_HIPBLAS=ON -DCMAKE_HIP_ARCHITECTURES="gfx803;gfx900;gfx906;gfx908;gfx90a;gfx1010;gfx1030;gfx1100;gfx1101;gfx1102" -DCMAKE_BUILD_TYPE=Release
make -j16 -C build
build/bin/llama-cli -ngl 32 --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -m ../dolphin-2.2.1-mistral-7b.Q5_K_M.gguf --prompt "Once upon a time"
¹ https://lists.debian.org/debian-ai/2024/07/msg00002.html ² Not including MI300 because that released too close to the Ubuntu 24.04 launch. ³ Pre-Vega architectures might work, but have known bugs for some applications. ⁴ Vega and RDNA 2 APUs might work with Linux 6.10+ installed. I'm in the process of testing that. ⁵ The version of rocBLAS that comes with Ubuntu 24.04 is a bit old and therefore lacks some optimizations for RDNA 3. It's also missing some MI200 optimizations.
Re: AMD GPU Inference
#63Earlier quoted context omitted.
So no doubt modern software is ridiculously bloated, but ROCm isn't just a GPU driver. It includes all sorts of tools and libraries as well. By comparison, if you go and download the CUDA toolkit as a single file, you get a download file that's over 4GB, so quite a bit larger than the download size you quoted. I haven't checked how much that expands to (it seems the ROCm install has a lot of redundancy given how well…
I suspected that, but any binaries being that large just seems wrong, I mean the whole thing is 35 time larger than my entire OS install. Do you know what is included in ROCm that could be so big? Does it include training datasets or something?
Here's the big files in my /opt/rocm/lib which is most of it:
4.8G hipblaslt
1.6G libdevice_conv_operations.a
2.0G libdevice_gemm_operations.a
1.4G libMIOpen.so.1.0.60200
1.1G librocblas.so.4.2.60200
1.6G librocsolver.so.0.2.60200
1.4G librocsparse.so.1.0.60200
1.5G llvm
3.5G rocblas
2.0G rocfft
There are mostly GPU kernels that by themselves aren't so big, but for every single operation x every single supported graphics architecture, eg:
304K TensileLibrary_SS_SS_UA_Type_SS_Contraction_l_Ailk_Bjlk_Cijk_Dijk_gfx942.co
24K TensileLibrary_SS_SS_UA_Type_SS_Contraction_l_Ailk_Bjlk_Cijk_Dijk_gfx942.dat
240K TensileLibrary_SS_SS_UA_Type_SS_Contraction_l_Ailk_Bljk_Cijk_Dijk_gfx942.co
20K TensileLibrary_SS_SS_UA_Type_SS_Contraction_l_Ailk_Bljk_Cijk_Dijk_gfx942.dat
344K TensileLibrary_SS_SS_UA_Type_SS_Contraction_l_Alik_Bljk_Cijk_Dijk_gfx942.co
24K TensileLibrary_SS_SS_UA_Type_SS_Contraction_l_Alik_Bljk_Cijk_Dijk_gfx942.datRe: AMD GPU Inference
#64Earlier quoted context omitted.
I suspected that, but any binaries being that large just seems wrong, I mean the whole thing is 35 time larger than my entire OS install. Do you know what is included in ROCm that could be so big? Does it include training datasets or something?
My understanding is that ROCm contains all included kernels for each supported architecture, so it would have (made up): -- matrix multiply 2048x2048 for Navi 31, -- same for Navi 32, -- same for Navi 33, -- same for Navi 21, -- same for Navi 22, -- same for Navi 23, -- same for Navi 24, etc. -- matrix multiply 4096x4096 for Navi 31, -- ...
Correct. Although, you wouldn't find Navi 22, 23 or 24 in the list because those particular architectures are not supported. Instead, you'd see Vega 10, Vega 20, Arcturus, Aldebaran, Aqua Vanjaram and sometimes Polaris.
We're working on a few different strategies to reduce the binary size. It will get worse before it gets better, but I think you can expect significant improvements in the future. There are lots of ways to slim the libraries down.
Re: AMD GPU Inference
#65Earlier quoted context omitted.
good feedback thanks, would you be able to open an issue
this repo? https://github.com/AUGMXNT/llm-tracker.info-vault/issues
I think fazkan was confused about which repo you were talking about. For the llm-tracker doc, that's something I maintain. It's based on stuff I test but if you want to submit a PR or issue w/ info in a way that I can verify then I'm happy to add a Docker section.
Re: AMD GPU Inference
#66Earlier quoted context omitted.
good feedback thanks, would you be able to open an issue
this repo? https://github.com/AUGMXNT/llm-tracker.info-vault/issues
haha, I was a bit confused, but I was referring to this one https://github.com/slashml/amd_inference. But the comment applies to other repos as well, do open issues in them, helps the maintainers prioritize features.
Re: AMD GPU Inference
#67Re: AMD GPU Inference
#68On Ubuntu 24.04 (and Debian Unstable¹), the OS-provided packages should be able to get llama.cpp running on ROCm on just about any discrete AMD GPU from Vega onwards²³⁴. No docker or HSA_OVERRIDE_GFX_VERSION required. The performance might not be ideal in every case⁵, but I've tested a wide variety of cards: # install dependencies sudo apt -y update sudo apt -y upgrade sudo apt -y install git wget hipcc libhipblas-de…
I was able to install (AMD provided) ROCm and Ollama on Ubuntu 22.04.5 with an RX 7900 XTX with no real problems to speak of, and I can execute LLMs using Ollama on ROCm just fine. Take that FWIW.
Re: AMD GPU Inference
#69Sad that RDNA2 cards aren't supported. Not even that old!
Re: AMD GPU Inference
#70So, this is all I needed to add to NixOS workstation:
hardware.graphics.enable = true;
services.ollama = {
enable = true;
acceleration = "rocm";
environmentVariables = {
ROC_ENABLE_PRE_VEGA = "1";
HSA_OVERRIDE_GFX_VERSION = "11.0.0";
};
};