71–80 of 86 posts
I find it funny that people say "only" for a setup of 64GB RAM and 8GB VRAM. That's a LOT. I'd have to spend thousands to get that setup.
I'm a little confused how these models run/fit onto VRAM. I have 32gb system RAM and 16gb VRAM. I can fit the 20b model all within vram, but then I can't increase the context window size past 8k tokens or so. Trying to max the context size leads to running out of VRAM. Can't it use my system ram as backup though? Yet I see other people with less resources like 10GB of vram and 32gb system ram fitting the 120b model o…
I find it funny that people say "only" for a setup of 64GB RAM and 8GB VRAM. That's a LOT. I'd have to spend thousands to get that setup.
https://frame.work/products/desktop-diy-amd-aimax300/configu... $1599 - $1999 isn't really a crazy amount to spend. These are preorder, so I'll give you that this isn't an option just yet.
Earlier quoted context omitted.
Given that this is at the middle/low-end of a consumer gaming setups - it seems particularly realistic that many people can run this out of the box on their home PC - or with an upgrade for a few hundred bucks. This doesn't require an A100 or some kind of fancy multi-gpu setup.
Not that these specs are outrageous, but “middle/low” is underselling it. The typical PC gamer has a modest system, despite all the noise from enthusiasts. The Steam hardware survey puts ~5% of people with 64GB RAM or more https://store.steampowered.com/hwsurvey
I find it funny that people say "only" for a setup of 64GB RAM and 8GB VRAM. That's a LOT. I'd have to spend thousands to get that setup.
I'm a little confused how these models run/fit onto VRAM. I have 32gb system RAM and 16gb VRAM. I can fit the 20b model all within vram, but then I can't increase the context window size past 8k tokens or so. Trying to max the context size leads to running out of VRAM. Can't it use my system ram as backup though? Yet I see other people with less resources like 10GB of vram and 32gb system ram fitting the 120b model o…
It seems you'll have to offload more and more layers to system RAM as your maximum context size increases. llama.cpp has an option to set the number of layers that should be computed on the GPU, whereas ollama tries to tune this automatically. Ideally though, it would be nice if the system ram/vram split could simply be readjusted dynamically as the context grows throughout the session. After all, some sessions may n…
Not a major setback because for long context I'd just use GPT or claude, but it would be cool to have 128k context locally on my machine. When I get a new CPU I'll upgrade RAM to 64, my GPU is more than capable of what I need for a while and a 5090 or 4090 is the next step up in VRAM but I don't want to shell out 2k for a card.
If you run these on your own hardware can you take the guard-rails off (ie "I'm afraid I can't assist with that"), or are they baked into the model?
Yes but the abliterated versions (those with partially removed guardrails) are significantly “dumber” so the trade off isn’t worthwhile imho.
Earlier quoted context omitted.
I would so appreciate concrete data instead of subjectivities like "excellent" and "super slow". How many tokens is excellent? How many is super slow? How many is non-filled context?
I'm not really timing it as I just use these models via open webui, nvim and a few things I've made like a discord bot, everything going via ollama. But for comparison, it is generating tokens about 1.5 times as fast as gemma 3 27B qat or mistral-small 2506 q4. Prompt processing/context however seems to be happening at about 1/4 of those models. A bit more concrete of the "excellent", I can't really notice any differ…
After every chat, open webui is sending everything to llamacpp again wrapped in a prompt to generate the summary, and this wipes out the KV cache, forcing you to reprocess the entire context.
This will get rid of the long prompt processing times id you're having long back and forth chats with it.