Viewing profile — throwdbaaway

Objectively speaking, the 2 bit quant from antirez has very low accuracy. Meanwhile, his 4 bit quant does have decent accuracy, but is a bit pointless by being bigger than the full…

They need to get a license from moonshot to provide inference for K3. Probably have to follow the pricing set by moonshot as well.

It works, thanks to https://github.com/ikawrakow/ik_llama.cpp/pull/1911 , which got merged in early June. However, there might still be some issue with the chat template.

I suspect this is why DeepSeek had to introduce the 2x peak hours pricing. The price would be too low otherwise.

Yeah antirez made a lot of big claims in that paragraph. Sounds like a case of AI psychosis.

If you max out the ram, TG with q3 should be at least 10 t/s. And with dsa, it can still stay close to that number as the context grows.

That's exactly what I said. They do care when FLOPs are involved. Restoring an old session with 900k tokens will require a lot of FLOPs to reprocess the 900k token. Meanwhile, they…

Different sessions. With https://github.com/fairydreaming/llama.cpp/tree/dsv4 , 1M context with DSV4 Flash takes less than 6GB of VRAM. I can't run DSV4 Pro, but it should take les…

Well I wouldn't call it a low bar, since some of the edits were quite complex. And 1M context in less than 6GB of VRAM is truly impressive, but somehow this gets way less attention…

While we are all speculating, Boris kindly provided some guidance in https://news.ycombinator.com/item?id=47880089 > The challenge is: when you let a session idle for >1 hour, when…

Indeed they are all lossy. Not sure how much they contribute to the quality loss in long context though. I got a 700k session with DSV4 Pro (official API), and the model was still …

The current top comment in https://lobste.rs/s/ua1gxl/glm_5_2_coming_ai_margin_collapse correctly zoomed into cached input tokens, but landed on the opposite conclusion: > That is,…

Seems like a pretty pointless post that still centers around output tokens. In agentic coding, cached input tokens is 90% of the API "cost". It doesn't require GPU compute, and Dee…

And somehow they claimed that it is "lossless".

On ZFS with zstd compression, I am getting 1.34x compressratio for the BF16 weights (across multiple models). Here's the du output for GLM-5.2: $ du -s -BG /cube/models/zai-org/GLM…

And their disk-based caching is amazing. I got a long 700k context session spanning more than a week, with pauses in between that was longer than a day, and some rewinds mixed in a…

Hah, that's because the prompt itself was only about 30 tokens. We need a much bigger prompt to properly test PP.

Huh that's not what I gathered from the tweet at all. If I am going to write a five why's analysis, the immediate cause is the LLM wrongly decided to delete a volume, while the roo…

If I understand correctly, both the staging database and the production database share the same volume. Thus, production data was gone as well after deleting the volume. 1st hint -…

Should be about 10~20 GiB per session. Save/restore is exactly what DeepSeek does using its 3FS distributed filesystem: https://github.com/deepseek-ai/3fs#3-kvcache With this much …

Based on the release schedule of 3.5 previously, my optimistic take is that they distill the small models from the 397B, and it is much faster to distill a sparse A3B model. Hopefu…

His Vibe Coding book is invaluable as a textbook example of slop.

https://github.com/anthropics/claude-code/issues/46829#issue... - Have you checked with your colleague? (and his AI, of course)

> EC2 instances on shared hardware showed up to 30% variance between runs due to noisy neighbors. Based on this finding, I suppose the better way is to rely on local hardware whene…

Very nice TG improvement from Flash Attention KQ fusion. Is it something that was already done in ik_llama.cpp? If not, then it will be a welcomed addition for hybrid CPU/GPU infer…