81–90 of 94 posts
Recently I had switched to OpenCode to try out many of the Non-US-Frontier-Labs models. My unexpected favorite model to use was Mercury (a diffusion model). Not because it was “smart” but because it was stupid fast. It was more of a pair-programming experience instead of the SOTA agentic experience of prompting and waiting. Honestly, it was also way more fun and brought back some of the pre-AI coding experience while…
I'm not getting anywhere near the speeds advertised on my 3090 Ti, alas, but it's fun watching it "fill out" its answers. I did Simon's "SVG pelican on a bicycle" test on it and the result was quite minimalistic but fit the brief: https://gist.github.com/peterc/7672e74ec1437945e5fca5ce2c1c9... -- this was on the Q4 quant running on patched llama.cpp. I will be interested to see if Simon's looks much different.
Some of these comments miss the advantage of diffusion. This is will have a big impact on edge devices, such as your phone or the GPU in your computer. An LLM's decoder computes tokens one-at-a-time because attention has to account for each previous token. The existing LLM decoders scale well when you have enough load to batch many inferences together. Diffusion of limited benefit there. On edge you have a different…
Edge devices don't just have limited memory bandwidth though, they also have very limited compute. To the extent where you don't actually need all that much batching to saturate their viable compute and run into obvious thermal/power limits. (It's just not true that "requests are inherently serial" in edge inference; any time you have multiple requests (i.e. "chats") in flight, batching becomes applicable if you have…
Earlier quoted context omitted.
YESSSS!!! speed is THE way! I like my boilerplate POJOs/data classes generated at breakneck pace of 300+ tok/s, Flash-Lite is more useful than GPT-5.5 for me this way. if it's too slow, you just stay in that goddamn async death loop
For boilerplate, yeah. But when asking research or exploratory questions, or weighing whether a feature is well designed, or asking "can I implement _x_ feature using these libraries without introducing unnecessary complexity", then GPT-5.5 medium is still fast enough. 10-20 seconds times a couple turns on a new feature isn't bad. Kimi is also similarly fast if not faster. I do agree with smaller models for more cons…
Earlier quoted context omitted.
YESSSS!!! speed is THE way! I like my boilerplate POJOs/data classes generated at breakneck pace of 300+ tok/s, Flash-Lite is more useful than GPT-5.5 for me this way. if it's too slow, you just stay in that goddamn async death loop
> I like my boilerplate POJOs/data classes generated at breakneck pace of 300+ tok/s Regardless of speed, use the LLM to eliminate the need for boilerplate rather than just creating more code faster. > if it's too slow, you just stay in that goddamn async death loop Things get slow when you're ballooning the size of your code, files, design and architecture, and things get more involved and complicated, piling fast h…
I'm not getting anywhere near the speeds advertised on my 3090 Ti, alas, but it's fun watching it "fill out" its answers. I did Simon's "SVG pelican on a bicycle" test on it and the result was quite minimalistic but fit the brief: https://gist.github.com/peterc/7672e74ec1437945e5fca5ce2c1c9... -- this was on the Q4 quant running on patched llama.cpp. I will be interested to see if Simon's looks much different.
Hi! What implementation are you using? Right now VLLM is the one recommended. llama.cpp is in an early draft
Earlier quoted context omitted.
For boilerplate, yeah. But when asking research or exploratory questions, or weighing whether a feature is well designed, or asking "can I implement _x_ feature using these libraries without introducing unnecessary complexity", then GPT-5.5 medium is still fast enough. 10-20 seconds times a couple turns on a new feature isn't bad. Kimi is also similarly fast if not faster. I do agree with smaller models for more cons…
well, I can usually think for myself or hit someone up in Discord (or Teams, if it's for a living) and in a worst case (that person just deflects to AI anyway) just save some token budget for myself
Every line I put in my app, I still reason about myself. But when deciding between 5+ ways of building some random, non-straightforward feature, it's nice to have what's essentially a "mentor" AI.
Earlier quoted context omitted.
This is a different model with, confusingly, approximately the same number of params as the existing gemma4 MoE. Unclear from a quick scan whether one was trained somehow from the other. The mechanism isn't the same as speculative decoding. Speculative decoding happens sequentially and (usually) a couple of tokens at a time; diffusion doesn't, and does blocks of text at once. I haven't read the collateral yet but my…
Thanks. I found this other comment that links to a very thorough explanation: https://news.ycombinator.com/item?id=48479042
Earlier quoted context omitted.
> I like my boilerplate POJOs/data classes generated at breakneck pace of 300+ tok/s Regardless of speed, use the LLM to eliminate the need for boilerplate rather than just creating more code faster. > if it's too slow, you just stay in that goddamn async death loop Things get slow when you're ballooning the size of your code, files, design and architecture, and things get more involved and complicated, piling fast h…
previously, mugging through docs to turn them into serializables for some API took weeks of grueling work if you wanted to cover an entire API surface that's as big as, say, GitHub's. nowadays, just "copy Markdown" from the very same GitHub, put 10-12 data classes, and let LLM extrapolate from there. with Gemini's 65.5k max token output, that is just several prompts and about two hours. that's the boilerplate. there…
Sounds like you're trying to just re-implement a HTTP API, not really boilerplate.
Boilerplate is code you could have avoided written, but you take the "temporary" shortcut of copy-pasting the code instead of building a proper abstraction. This is what I'm talking about is the wrong direction.