161–170 of 227 posts
I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
Earlier quoted context omitted.
agreed!! in my heart i really wanted to say by the end of 2026 but wanted to add some wiggle room in case they start to ban open source AI development.
>in case they start to ban open source AI development Is that likely, do you think?
Earlier quoted context omitted.
This reads as pretty clearly AI-generated text, which is against HN guidelines.
The PR? He said it was AI in the comment you replied to... I don't think the post itself reads like AI at all, but that's just me.
https://www.pangram.com/history/61fbc90e-180d-4d91-a85d-16ff...
Earlier quoted context omitted.
I'm sure this exact topic has been argued hundreds of times already on HN, but I think I have a new "possibly agreeable to both sides" perspective on this after having lost man-years to retired corporate code aka "FAIAP, throwaway code" Let LLMs write the corpo code, as it will be unlikely to still be running in 5-10 years. Frontier AI is already at the point where it writes fewer bugs per LOC than humans. By a lot.…
There's a whole spectrum of employment between faceless corporations and personal side projects. AI will replace humans because giant business believe they can do the same work, not because they will actually be able to. The correctness of an application is limited by your ability to understand and describe what you need. We have a word for an application specification tool so detailed it eliminates all ambiguity. It…
This has been being claimed for at least 2 years now. Wouldn't we already be seeing disasters if this was the case? It's certainly been around long enough to cause some real damage. Instead there is a slow trickle of things that makes the news by people who didn't bother instituting a single fucking control. For example, I am completely immune to `rm -rf` style fuckups because I wrote rm-safe years ago and it is mapped to `rm` in any environment that the LLM will run in: https://github.com/pmarreck/rm_safe
In any event, it's not a replacer, it's an augmenter. Nothing will replace humans, because we are the stakeholders; it may shift them around, though.
> The correctness of an application is limited by your ability to understand and describe what you need.
If you write code in Lean 4 or Idris 2, you may not completely understand why it is or isn't correct, but their respective compilers will certainly prove it to you one way or the other.
We already are perfectly functional with incomplete understandings. An entire generation of web developers have had successful careers without having a single clue how compilers work or how machine code works. Now is there a type of problem that sometimes comes along that DOES require deep understanding? Of course, that's when you call in the heavy artillery. Or the advanced LLM. Or both. See my point?
Author here. The short version: a viral post ran Gemma 4 on a 2016 Xeon; my Xeons are 2013, and the fork it used assumes AVX2, which Ivy Bridge doesn't have. The build failure was easy. The fun bug was the silent one: two MoE graph ops with no dispatch case on non-AVX2 builds, so every expert FFN output was uninitialized memory. Deterministic, NaN-free, fluent-looking multilingual gibberish. The fix is open upstream…
https://www.pangram.com/history/61fbc90e-180d-4d91-a85d-16ff...
Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally. If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider. I estimate that the server consumes probably around 500W during inference. In Germany where 1kwh cost arou…
I run qwen 27b at home when working it pulls around 400W. I get 40ish tokens per second generation and more importantly about 1000 tokens per second prompt processing. In an hour it can process 3.6 million tokens or generate 144000 tokens. This costs me about 15 cents given my electricity prices. For sonnet the equivalent token costs are 7.2 dollars for the prompt processing or 1.4 dollars for the generation. The clo…
My guess is that this math gets less good with MoE (because you will be limited by VRAM, but clouds won't).
Earlier quoted context omitted.
We're too easily conflating parameter count with capability. That Bonsai 27B you're running is at 2-bit quantization. Is it really better than the best 10-18B models?
1. yes. https://www.alphaxiv.org/abs/2607.bonsai-27b table 14 shows that bonsai retains roughly 95% of the fp16 27b model's average performance and outperforms post-training quantization at a similar bit width. it doesn't directly compare against every top 10-18b model, but it is clearly still performing like a large model. 2. quantization != native low precision training. a model trained in native ternary should gen…
I know that's what the paper says the benchmarks say, but these models feel significantly worse than the base model when you start using them for real tasks.
Even the Q4 quant which they put in between their Bonsai models and the FP16 in the benchmarks has a tendency to go into doom loops and get lost compared to even Q5 or Q6.
I don't know how much of this is due to benchmaxxing (putting the benchmarks into the post-training loop) or cherry picking benchmarks to look good. If you spend a lot of time using local models you learn to take vendor provided benchmarks with a huge heap of doubt. Everything looks amazing in the benchmarks these days.
I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
Isn’t 9 tok/s unbearable? I frequently see Claude sessions hit 1Mtok in less than a day. It seems that 9tok/s would be really slow for actual work.
Earlier quoted context omitted.
First off the math doesn’t math. Datacenters are willing to pay $50k for a single high end GPU. If you have unlimited capacity, yeah sell millions for $100 a pop or $10 a pop or whatever the bom cost of a phone GPU would be - but if you have limited capacity, you’re gonna sell all of that to the customer who is willing to pay the most PER UNIT. Second off, this doesn’t work from a power consumption standpoint. When I…
> Datacenters are willing to pay $50k for a single high end GPU. its true for now, because capital is flowing like a torrent, but how long will that last if returns start to be expected (aka the bubble pops)?
That doesn’t change until production capacity exceeds the datacenter demand. When that happens, they’ll start selling them down the market until it eventually reaches phones and toasters and whatever. But not in two years.