341–345 of 345 posts
Earlier quoted context omitted.
There was a startup posted here which built custom hardware that let the AI respond instantly. Thousands of tokens per second.
Taalas. A sibling comment of yours posted the chat demo URL - https://chatjimmy.ai/
I.e.: burn the weights into resistors with a range of possible values, and do the sums through simply adding up the currents along parallel paths by simply connecting them!
Earlier quoted context omitted.
There is a newer PR which will probably be merged soon: https://github.com/ggml-org/llama.cpp/pull/22673
Ohhhh geee!!! I just applied the patch to my local git copy. You need to use the model on the PR that he submitted, the model is particular because it has extra information that allows the MTP to happen. I have two amd gpus, and qwen3.6 27B qk6 does around 20t/s generation... If I run it only on one I get like 35t/s. But with this patch I saw 46t/s with qwen3.6 27B q8... this is insane, it's 250% faster than the orig…
Earlier quoted context omitted.
The small draft model proposes a sequence of tokens d1 d2 d3. The big target model calculates P(d1) P(d2|d1) P(d3|d1 d2) In parallel. If we were just greedy decoding it would be simple. Just stop when the draft model doesn’t predict the most likely token as judged by the target model. At that point, append the correct token from the target model and kick off both models again in parallel. In practice we aren’t using…
nice ... i think i get the idea - it's effectively the same / similar benefit as batching, but you're batching against your own speculated future path. Which would be pointless if you didn't have a high probability path to evaluate against - but the draft gives you that.
MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 ) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.
I don’t exactly know where MTP inference fits within the inference stack, but does someone know whether it’s possible to implement it for the MLX universe?
Earlier quoted context omitted.
You can use it for free with Google AI studio (free tier or paid tier accounts with different limits). Or use the paid version from Vertex AI which is around 3x cheaper than Gemini 3 Flash. I'm using Gemma 4 31B in my app with 5 agents, 1.5k requests per day, each.
I'm curious what tasks you use this model for?
You can use it for free, forever, if you just run the bot in your browser (client mode). Server mode is premium, but you don't need it to run the bots.
I posted about it in this comment: https://news.ycombinator.com/item?id=48085993#48088468