[flagged]
Personally I'm hoping for EU to step up a bit
Sorry we a focusing on leather handbags there.
DeepSeek V4 Flash 0731 Intelligence, Performance and Price Analysis
151–160 of 342 posts
Re: DeepSeek V4 Flash 0731 Intelligence, Performance and Price Analysis
#152New Deepseek models are like Christmas for me. Really big fan of low cost API models, noone does it better than DS. Until VRAM price is low enough to run models locally, this is the way to go. The subsidized subscription model won't last, API pricing "feels" closer to a true sustainable business model.
Indeed. My fellow software engineers keep complaining about using up all their Claude tokens within an hour... Whilst I'll be rocking DS flash for the entire day. Sure it gets a few things wrong here and there, but that's when you pull out the Claude models or whatever for those tricky tasks.
The problem is picking between models. I do not want to spend my time switching models and trying to decipher which model should be used for what. Maybe that's just a me problem that I need to figure out.
Re: DeepSeek V4 Flash 0731 Intelligence, Performance and Price Analysis
#153Earlier quoted context omitted.
I'm neither pro China, nor pro US. I'm pro open weights models, and I'm pro cheaper hardware. At this point I don't see any american frontier labs releasing SOTA open weights model, and I don't see ASML/Nvidia/Samsung monopoly getting any competition from anywhere apart from China in the near future.
> I'm neither pro China, nor pro US. I'm pro open weights models, and I'm pro cheaper hardware. yea i got that from your first comment ( although you removed crush American companies in _price_ ). you are pro cheapness at any cost even if its from your country's state funded direct geopolitical enemy. China can always count on first order greed to win
Why does them being a geopolitical enemy matter to me as an individual?
Re: DeepSeek V4 Flash 0731 Intelligence, Performance and Price Analysis
#154The weights were just released a few minutes ago: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731
Hmm, it's targeting a HW accelerator with a 128x128 matmul primitive, which one is that? Warp Group Matrix Multiply Accumulate on H100?
Blackwell. You could do it with wgmma on hooper but then you'll only be able to run 1 CTA per SM. There are cases where this is OK, but most commonly 128x128 mmas are primitive in tcgen05.mma i.e blackwell. The fundamental reasons is that the systolic array accelerator (TMA) until blackwell wrote to the cuda core registers themselves so you were limited by the register file size of the SM[SP]. In blackwell there's a separate TMEM where the mma unit stores it's output.
You can even do higher 256x256 and such using a hardware feature called 2-CTA MMA, this is essentially them letting neighbouring pairs of SMs co-operate and access each other's memory.
As for sibling comments, huawei ascend is more of an NPU-style architecture where you can easily have much bigger MMAs as primitive. But you usually don't anyways for many reasons.
Re: DeepSeek V4 Flash 0731 Intelligence, Performance and Price Analysis
#155Re: DeepSeek V4 Flash 0731 Intelligence, Performance and Price Analysis
#156Earlier quoted context omitted.
What kind of tps are you getting?
Generally get 20-25tps - prefill is pretty good around 400-450tps. I have been using compaction at around 100k tokens but mostly just cause it was the default in pi coding agent - might see if i can expand it a bit.
I've had it run to ~400k when debugging "obscure" (to it) sequences. Wouldn't recommend more.
Re: DeepSeek V4 Flash 0731 Intelligence, Performance and Price Analysis
#157Re: DeepSeek V4 Flash 0731 Intelligence, Performance and Price Analysis
#158[flagged]
I'm not looking forward to it, them being strapped for resources provides a huge incentive to develop and release these smaller models. Even if they'd still release their models once they are able to comfortably service all potential customers via their cloud, running them locally would be almost impossible due to their size.
Chinese are pretty pragmatic. Even if they produce more expensive chips and memory, they are still going to focus on value. But who knows, maybe India will step up again like they did for Y2K 3 decades ago.
Re: DeepSeek V4 Flash 0731 Intelligence, Performance and Price Analysis
#159Earlier quoted context omitted.
Hmm, it's targeting a HW accelerator with a 128x128 matmul primitive, which one is that? Warp Group Matrix Multiply Accumulate on H100?
Just because scales are grouped by 128x128 tiles, does not mean you need a single compute tile that large. It works completely fine to process it with multiple smaller tiles that get given the same scales, like how this works on Hopper and Blackwell today
[deleted]
Re: DeepSeek V4 Flash 0731 Intelligence, Performance and Price Analysis
#160For people with single RTX PRO 6000 96GB or DGX Spark 128GB, vllm-moet is a very good engine, although lesser known. It auto generate a symmetric 2-bit plane for inference and also generate a 4-bit delta cache to recover precision. Support ssd streaming oversized weight. You pick how much VRAM to allocate to each to balance out speed vs precision. 170 tps with ds-v4-flash demonstrated.
It use the stock model, no new models requires.
Worth spend a few hours to try.
The DGX Spark requires a small hack to ignore the difference between sm120 vs sm121, but it does run on sm121.