31–40 of 234 posts
Earlier quoted context omitted.
What does performance look like?
Spoiler alert: not good enough to break CUDA's moat
Inference side is partly about performance, but mostly about cost per token.
And given that there has been a ton of standardization around LLaMA architectures, AMD/ROCm can target this much more easily, and still take a nice chunk of the inference market for non-SOTA models.
Feels like all of this local LLM stuff is definitely pushing people in the direction of getting new hardware, since nothing like RX 570/580 or other older cards sees support. On one hand, the hardware nowadays is better and more powerful, but on the other, the initial version of CUDA came out in 2007 and ROCm in 2016. You'd think that compute on GPUs wouldn't require the latest cards.
I'm a happy user of Mistral on my Mac Air M1.
I wonder why they aren't supporting RX 6750 XT and lower yet, are there architectural differences between these and RX 6800+?
https://github.com/ollama/ollama/issues/2870#issuecomment-19...
I wish AMD did well in the Stable Diffusion front because AMD is never greedy on VRAM. The 4060Ti 16GB(minimum required for Stable Diffusion in 2024) starts at $450. AMD with ROCm is decent on Linux but pretty bad on Windows.
They bump up VRAM because they can't compete on raw compute.
Earlier quoted context omitted.
I'm a happy user of Mistral on my Mac Air M1.
Is it easy to set this up?
It's mostly optimized for M-series silicon, but it also technically works on Windows, and isn't too difficult to trick into working on Linux either.
Earlier quoted context omitted.
Is it easy to set this up?
Super easy. You can just head down to https://lmstudio.ai and pick up an app that lets you play around. It's not particularly advanced, but it works pretty well. It's mostly optimized for M-series silicon, but it also technically works on Windows, and isn't too difficult to trick into working on Linux either.
Feels like all of this local LLM stuff is definitely pushing people in the direction of getting new hardware, since nothing like RX 570/580 or other older cards sees support. On one hand, the hardware nowadays is better and more powerful, but on the other, the initial version of CUDA came out in 2007 and ROCm in 2016. You'd think that compute on GPUs wouldn't require the latest cards.