41–50 of 55 posts
Earlier quoted context omitted.
I find these comments hilarious. Are we supposed to build AI and then not use it? Super goofy.
Not "not use it". Not use it to make people believe they're talking to real people.
This is a Claude-code generated repo that implements some ideas from research papers. If you follow this space, every paper release spawns tens or hundreds of vibecoded repos like this that get spammed to Reddit, Hacker News, and other sites. It's generally best to overlook the vibecoded repos and go closer to the source for up to date information. In this case, z-lab already showed Qwen3.5-27B with DFlash last month…
This reads like you didn’t read the post. z-lab runs BF16 on B200 (54+ GB). There is no z-lab path that fits on a 24 GB 3090. That is literally the entire point of our work, and it is stated in the second paragraph. If you had checked the HF model card you linked before posting, you would see the same thing. Before this repo, there was no path to run this... SGLang's GGUF path for this model is broken. llama.cpp does…
Earlier quoted context omitted.
AI authored comments are against the rules. that said what is the point of engaging here if you won't do it with your own words? like do you understand any of what you wrote?
I find these comments hilarious. Are we supposed to build AI and then not use it? Super goofy.
These are the rules.
Earlier quoted context omitted.
That's a pretty popular budget friendly GPU people use for local AI, it actually seems like an excellent choice IMHO.
Depends on your definition of budget friendly, I suppose. I was looking around the other day and the cheapest working 24GB RTX 3090 on eBay was $1800 CAD after exchange rate, shipping and all the rest. Hugely inflated from the $700 they were once going for. Maybe there are still deals around.
With one you can run 9B/12B models which are fine for text tasks like chatting or summarisation. Not for precision like tool calling or code.
With two of them you can run models up to Qwen 27B and 35B with a few-turn context window (8k-16k). Dense at 14t/s and MoE at 68t/s.
With three of them you can run 128k context, though you'll need a large format case and the right motherboard or PCIe riser.
I'm running three and even with a new case this setup cost me less than one 3090.
Earlier quoted context omitted.
That's a pretty popular budget friendly GPU people use for local AI, it actually seems like an excellent choice IMHO.
The only thing that compares to this is probably Mac mini with MLX models.
The local inference space is pretty good nowadays.
Why don't we have LLM-Accelerator cards (yet)? Just seems like graphics cards are a good choice, but not a purpose-built choice.
Earlier quoted context omitted.
Depends on your definition of budget friendly, I suppose. I was looking around the other day and the cheapest working 24GB RTX 3090 on eBay was $1800 CAD after exchange rate, shipping and all the rest. Hugely inflated from the $700 they were once going for. Maybe there are still deals around.
Actually budget friendly is RTX 3060 12Gb. With one you can run 9B/12B models which are fine for text tasks like chatting or summarisation. Not for precision like tool calling or code. With two of them you can run models up to Qwen 27B and 35B with a few-turn context window (8k-16k). Dense at 14t/s and MoE at 68t/s. With three of them you can run 128k context, though you'll need a large format case and the right moth…
"Local AI should be a default, not a privilege: private data, no per-token bill, no vendor lock-in. The hardware to run capable models already sits on desks. The software to run those chips well doesn't." So figure out how to run it on Vulkan instead of requiring the user to be locked into expensive CUDA cards.
So everyone is aware, you can already run Qwen3.5-27B on Vulkan or Apple's hardware. Every major inference engine supports it right now. This repo is a vibecoded demo implementation of some recent research papers combined with some optimizations that sacrifice quality for speed to get a big number that looks impressive. The 207 tok/s number they're claiming only appears in the headline. The results they show are half…
I just find it funny they talk about being vendor locked, and the only thing they support is nvidia.