71–80 of 184 posts
Earlier quoted context omitted.
> You would need multiple GPUs with shared memory if you wanted to offload the higher precision models to VRAM. Or just a powerful apple silicon machine? I've tried dolphin mixtral 4bit on a 36gb ram MacBook m3, and inference is super fast.
Did you do anything special to make that work? Is it useful? Or just a toy?
Earlier quoted context omitted.
Or a Linux machine with a Ryzen using the internal GPU and the unified RAM (scroll down at llama.cpp and look for ROCm).
What? I can do this? Runs to the PC EDIT: I cannot, I need to install ROCm to compile with it, and then install something called hipBLAS, and who knows what else.
What are good API providers that serve mixtral? I know only octo ai which seems decent but will be good to know alternatives too
Earlier quoted context omitted.
What? I can do this? Runs to the PC EDIT: I cannot, I need to install ROCm to compile with it, and then install something called hipBLAS, and who knows what else.
OpenCL should also work on AMD cards, and is way easier to install
EDIT: Oh, no, I have an nVidia GPU, AMD CPU.
Interesting, I must have missed the first Leo announcement. I really like how privacy conscious it is. They don’t store any chat record which is what I want.
Earlier quoted context omitted.
> You would need multiple GPUs with shared memory if you wanted to offload the higher precision models to VRAM. Or just a powerful apple silicon machine? I've tried dolphin mixtral 4bit on a 36gb ram MacBook m3, and inference is super fast.
Did you do anything special to make that work? Is it useful? Or just a toy?
Earlier quoted context omitted.
I can run 4bit on a beat up 1070 ti. GP talks about higher precision models
You wouldn’t be able to fit the whole model into 8GB VRAM. It’s faster than not using a GPU at all, but most of it would still be computed on the CPU.
Earlier quoted context omitted.
I prefer koboldcpp over llama.cpp. It’s easy to spilt between gpu/cpu on models larger than VRAM
I've got an aging 2080Ti and Ryzen 3800X with 96GB RAM, any point in trying to mess with the GPU or? Haven't really been able to justify upgrading to a 4090 or similar given I play so few new games these days.
Earlier quoted context omitted.
> You would need multiple GPUs with shared memory if you wanted to offload the higher precision models to VRAM. Or just a powerful apple silicon machine? I've tried dolphin mixtral 4bit on a 36gb ram MacBook m3, and inference is super fast.
Or a Linux machine with a Ryzen using the internal GPU and the unified RAM (scroll down at llama.cpp and look for ROCm).