31–40 of 102 posts
This seems to be some AI generated wrapper around a wrapper of a wrapper. > # Other AMD-specific optimizations can be added here > # For example, you might want to set specific flags or use AMD-optimized libraries What are we doing here, then?
Earlier quoted context omitted.
It's not just you; AMD manages to completely shit-up the Linux kernel with their drivers: https://www.phoronix.com/news/AMD-5-Million-Lines
> Of course, much of that is auto-generated header files... A large portion of it with AMD continuing to introduce new auto-generated header files with each new generation/version of a given block. These verbose header files has been AMD's alternative to creating exhaustive public documentation on their GPUs that they were once known for.
I almost tried to install AMD rocm a while ago after discovering the simplicity of llamafile. sudo apt install rocm Summary: Upgrading: 0, Installing: 203, Removing: 0, Not Upgrading: 0 Download size: 2,369 MB / 2,371 MB Space needed: 35.7 GB / 822 GB available I don't understand how 36 GB can be justified for what amounts to a GPU driver.
By comparison, if you go and download the CUDA toolkit as a single file, you get a download file that's over 4GB, so quite a bit larger than the download size you quoted. I haven't checked how much that expands to (it seems the ROCm install has a lot of redundancy given how well it compresses), but the point is, you get something that seems insanely large either way.
Does it work with an APU? I just put 64GB in my system and gonna drop in a 5700G. Will that be enough? SFF inference if so.
BTW if you just want to play with a local LLM, you can try my old port of Mistral: https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... Unlike CUDA or ROCm my port is based on Direct3D 11 GPU API, runs on all GPUs regardless of the brand.
What's the best bang-for-your-buck AMD GPU these days? I just bought 2 used 3090s for $750ish refurb'd on eBay. Curious what others are using for running LLMs locally.
I almost tried to install AMD rocm a while ago after discovering the simplicity of llamafile. sudo apt install rocm Summary: Upgrading: 0, Installing: 203, Removing: 0, Not Upgrading: 0 Download size: 2,369 MB / 2,371 MB Space needed: 35.7 GB / 822 GB available I don't understand how 36 GB can be justified for what amounts to a GPU driver.
So no doubt modern software is ridiculously bloated, but ROCm isn't just a GPU driver. It includes all sorts of tools and libraries as well. By comparison, if you go and download the CUDA toolkit as a single file, you get a download file that's over 4GB, so quite a bit larger than the download size you quoted. I haven't checked how much that expands to (it seems the ROCm install has a lot of redundancy given how well…
Do you know what is included in ROCm that could be so big? Does it include training datasets or something?
People use "Docker-based" all the time but what they mean is that they ship $SOFTWARE in a Docker image. "Docker-based" reads, to me, as if you were doing Inference on AMD cards with Docker somehow, which doesn't make sense.
People use "Docker-based" all the time but what they mean is that they ship $SOFTWARE in a Docker image. "Docker-based" reads, to me, as if you were doing Inference on AMD cards with Docker somehow, which doesn't make sense.
Hi, we (ZML), fix that: https://github.com/zml/zml
Does it work with an APU? I just put 64GB in my system and gonna drop in a 5700G. Will that be enough? SFF inference if so.
You just need to update the version check here
https://github.com/slashml/amd_inference/blob/4b9ec069c4b2ac...
feel free to open an issue, with the requirements and we will test it.