281–290 of 393 posts
DeepSeek is, as I feel currently, the sole AI company which is actually trying to innovate rather than top mere benchmarks. Others like OpenAI, Anthropic and Google are mostly just competeing with each rather than keep innovating around the clock.
the big labs have already been doing this for at least a year
Earlier quoted context omitted.
US labs do it too.
Name any 2 or 3 that published bleeding edge research and similar in the last 6 months. Well I can't think of even one at the moment, to be honest might be biased but all Chinese research labs are largely oss except Alibaba now. I am certain there are lots of American labs that claim to do it, but either they are marketting in hype since they aren't even close to the frontier or contrarily just don't make anything of…
Earlier quoted context omitted.
There was a recent exodus from Qwen of researchers who supported their open source efforts, I’m not sure we will see many new open models from them past the 3.6 series.
are you able to say more about this (specifically that the researchers who left were concerned that qwen models were being closed sourced)? I was under the impression that the chinese labs and their employees aren't particularly concerned about ideology/safety/whatever and were just releasing open source models because it helps with publicity and does the most damage to the US AI labs. (and to be clear, I strongly su…
Earlier quoted context omitted.
>publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Google is still releasing a lot of llm architecture research. They introduced speculative decoding of LLMs in 2022[1], then released the code to perform sceculative decoding for their Gemma 4 model this year[2] [1] https://arxiv.org/abs/2211.17192 [2] https://github.com/google-gemma/cook…
They weren't the first to do MTP like this, and arguably did it wrong: the MTP heads are kept in a separate file and have to be welded in by the inference engine. Qwen 3.6 shipped with working MTP first, and had working MTP in llama.cpp first.
Earlier quoted context omitted.
The idea that America is going to stay ahead of China is I think at this point clearly delusional. It's also just such silly framing. Why should 350 million people stay ahead of 1 billion people on the other side of the world? If an AI lab in China cures cancer or something do Americans lose? So many Americans seem to (at least in theory) be ready to sign up for this ongoing confrontation with China. Does anyone thin…
By this population-only logic, you should concede that India will overtake China. Why not talk about how China shut out American companies for decades before complaining about BYD? As an Indian immigrant, the PRC China has engaged in conflict with almost all its neighbors and stated wars in its short history. China is not so benevolent when they get to the #1 spot: https://m.economictimes.com/industry/renewables/chin…
As for the rest of it:
Earlier quoted context omitted.
I hate to quote pithy proverbs, but "the road to hell is paved with good intentions." One can have an altruistic goal which ends up harming people too, which is where that proverb comes from. Prohibition and The War on Drugs in the US are two good examples of something that had altruistic origins[†] but ended up doing way more harm than good. [†] Another problem with altruism: we don't all agree on whether a goal is…
> Prohibition and The War on Drugs in the US are two good examples of something that had altruistic origins The War on Drugs had the purpose (not just in its origin but in its perpetuation) of inflicting harm on elite-disfavored subsets of the population that could not be openly targeted for Constitutional reasons, which is about as far from an altruistic reason as it possible to get.
I’ve been using DeepSeek v4 pro for a month now in Kilo Code and its great. Fast, reliable, large context window and cheap as… Did 1,5B tokens this month and cost me 40usd (majority cached, but still).
I've been using omp with deepseek as my task and quicktask agents, and sonnet as everything else. It's drastically reduced my AI spend. I went from spending $40/day to $10/day.
Earlier quoted context omitted.
MoE is from Google (Noam Shazeer) MTP is from Meta Another DeepSeek advance that the west are copying is DeepSeek Sparse Attention (DSA)
Mixture-of-Expert (MoE) was introduced in the 1990s [1, 2], see also [3, 4]. The idea was that MoE scales up model capacity and only introduces small computation overhead. MoEs did not become viable for high-performance applications until sparse routing was integrated with modern deep networks, made possible by large-scale distributed computation. The breakthrough came with the development of sparsely gated networks…
Earlier quoted context omitted.
They weren't the first to do MTP like this, and arguably did it wrong: the MTP heads are kept in a separate file and have to be welded in by the inference engine. Qwen 3.6 shipped with working MTP first, and had working MTP in llama.cpp first.
Given the MTP drafter is basically a separate model, keeping it separate makes more sense IMO. It's out of my wheelhouse but it seems like you could adjust the MTP drafter model separately from the main model, too. Ultimately though the real explanation, I think, is Google doesn't care since for their own purposes (in LiteRT-LM), they do bundle them. As far as I know, anyway.
They are more like a single model that has two separate attention head mechanisms.
DeepSeek is, as I feel currently, the sole AI company which is actually trying to innovate rather than top mere benchmarks. Others like OpenAI, Anthropic and Google are mostly just competeing with each rather than keep innovating around the clock.
Please explain how distillation == innovation Especially since your 5-day-old account is sus, and thus likely not yet proven not to be a Chinese bot You can't lead by following the actual leader LOL The only real innovation I've seen from Deepseek is the out-loud reasoning thing in R1