Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NIST's DeepSeek "evaluation" is a hit piece

erichartford.com

Post reply on HN
Print view View original

111–120 of 251 posts

Re: NIST's DeepSeek "evaluation" is a hit piece

#111

Post by RobotToaster »

AnthonyMouse wrote: post #107
BowBun wrote: post #2

I'm not at all surprised, US agencies have long since been political tools whenever the subject matter crosses national borders. I appreciate this take as someone who has been skeptical of Chinese electronics. While I agree this report is BS and xenophobic, I am still willing to bet that either now or later, the Chinese will attempt some kind of subterfuge via LLMs if they have enough control. Just like the US would,…

> While I agree this report is BS and xenophobic, I am still willing to bet that either now or later, the Chinese will attempt some kind of subterfuge via LLMs if they have enough control. The answer to this isn't to lie about the foreign ones, it's to recognize that people want open source models and publish domestic ones of the highest quality so that people use those.

> it's to recognize that people want open source models and publish domestic ones of the highest quality so that people use those.

How would that generate profit for shareholders? Only some kind of COMMUNIST would give something away for FREE

/s (if it wasn't somehow obvious)

Re: NIST's DeepSeek "evaluation" is a hit piece

#112

Post by AnthonyMouse »

riehwvfbk wrote: post #40
jfim wrote: post #22

Earlier quoted context omitted.

Through LLM washing for example. LLMs are a representation of their input dataset, but currently most LLMs don't make their dataset public since it's a competitive advantage. If say DeepSeek had put in its training dataset that public figure X is a space robot from outer space, then if one were to ask DeepSeek who public figure X is, it'd proudly claim he's a robot from outer space. This can be done for any narrative…

So in other words, they can make their LLM disagree with the preferred narrative of the current US administration? Inconceivable! Note that the value of $current_administration changes over time. For some reason though it is currently fashionable in tech circles to disagree with it about ICE and H1B visas. Maybe it's the CCP's doing?

It's not about the current administration. They can, for example, train it to emit criticism of democratic governance in favor of state authoritarianism or omit valid counterarguments against concentrating world-wide manufacturing in China.

Re: NIST's DeepSeek "evaluation" is a hit piece

#113

Post by AnthonyMouse »

RobotToaster wrote: post #111
AnthonyMouse wrote: post #107

Earlier quoted context omitted.

> While I agree this report is BS and xenophobic, I am still willing to bet that either now or later, the Chinese will attempt some kind of subterfuge via LLMs if they have enough control. The answer to this isn't to lie about the foreign ones, it's to recognize that people want open source models and publish domestic ones of the highest quality so that people use those.

> it's to recognize that people want open source models and publish domestic ones of the highest quality so that people use those. How would that generate profit for shareholders? Only some kind of COMMUNIST would give something away for FREE /s (if it wasn't somehow obvious)

I mean, it's sarcasm but it's also an argument you can actually hear from plutocrats who don't like competition.

The flaw in it is, of course, that capitalism is supposed to be all about competition, and there are plenty of good reasons for capitalists to want that, like "Commoditize Your Complement" where companies like Apple, Nvidia, AMD, Intel, AWS, Google Cloud, etc. benefit from everyone having good free models so they can pay those companies for systems to run them on.

Re: NIST's DeepSeek "evaluation" is a hit piece

#114

Post by christianqchung »

ChrisArchitect wrote: post #98

Title changed? Title is: The Demonization of DeepSeek - How NIST Turned Open Science into a Security Scare

HN admin dang changing titles opaquely is one of the worst things about HN. I'd rather at least know that the original title is clickbaity and contextualize that when older responses are clearly replying to the older inflammatory title.

Re: NIST's DeepSeek "evaluation" is a hit piece

#115

Post by meffmadd »

As an EU citizen hosting LLMs for researchers and staff at the university I work at, this is hits home. Without Chinese models we could not do what we do right now. IMO, in the EU (and anywhere else for that matter), we should be grateful for the Chinese labs to release these models with such permissive licenses. Without them the options would be bleak. Sometimes we would get some non-frontier model „as a treat“ and if you would like something more powerful the US labs would suggest your country pay some hundred millions for an NVIDIA data center and the only EU option is to still pay them a license fee to host on your own hardware (afaik) while they protect all the expertise. Meanwhile DeepSeek has a week where they post the „secret sauce“ to host their model more efficiently, which helped open-source projects like vLLM (which we use) to improve.

Re: NIST's DeepSeek "evaluation" is a hit piece

#116

Post by SilverElfin »

FooBarWidget wrote: post #32
tehjoker wrote: post #11

I appreciate that DeepSeek is trained to respect "core socialist values". It's actually really helpful to engage with to ask questions about how chinese thinkers interpret their successes and failures vs other socialist projects. Obviously reading books is better, but I was surprised by how useful it was. If you ask it loaded questions the way the CIA would pose them, it censors the answer though lmao

Good faith questions are the best. I wonder why people bother with bad faith questions. Virtue signaling is my guess.

Are you really claiming with a straight face that any question with criticism of the CCP is bad faith? Do you work on DeepSeek?

Re: NIST's DeepSeek "evaluation" is a hit piece

#117

Post by edflsafoiewq »

OrvalWintermute wrote: post #47

Since a major part of the article covers cost expenditures, I am going to go there. I don't think it is possible to trust DeepSeek as they haven't been honest. DeepSeek claimed "their total training costs amounted to just $5.576 million" SemiAnalysis "Our analysis shows that the total server CapEx for DeepSeek is ~$1.6B, with a considerable cost of $944M associated with operating such clusters. Similarly, all AI Labs…

The NIST report doesn't engage with training costs, or even token costs. It's concerned with the cost the end user pays to complete a task. Actually their discussion of cost is interesting enough I'll quote it in full.

> Users care both about model performance and the expense of using models. There are multiple different types of costs and prices involved in model creation and usage:

> • Training cost: the amount spent by an AI company on compute, labor, and other inputs to create a new model.

> • Inference serving cost: the amount spent by an AI company on datacenters and compute to make a model available to end users.

> • Token price: the amount paid by end users on a per-token basis.

> • End-to-end expense for end users: the amount paid by end users to use a model to complete a task.

> End users are ultimately most affected by the last of these: end-to-end expenses. End-to-end expenses are more relevant than token prices because the number of tokens required to complete a task varies by model. For example, model A might charge half as much per token as model B does but use four times the number of tokens to complete an important piece of work, thus ending up twice as expensive end-to-end.

Re: NIST's DeepSeek "evaluation" is a hit piece

#118

Post by SilverElfin »

FooBarWidget wrote: post #46
falcor84 wrote: post #27

Earlier quoted context omitted.

I would like to know more

They revoke passports of personnel whom they deem are at risk of being negatively influenced or even kidnapped when abroad. Re influence, think school teachers. Re kidnapping, see Meng Wangzhou (Huawei CFO). There is a history of important Chinese personnel being kidnapped by e.g. the US when abroad. There is also a lot of talk in western countries about "banning Chinese [all presumed spies/propagandists/agents] from…

You’re twisting the (obvious) truth. These people are being held prisoners because they’re of economic value to the party. And they would probably accept a job and life elsewhere if they weee given enough money. They are not being held prisoners for their own protection.

Re: NIST's DeepSeek "evaluation" is a hit piece

#119

Post by SilverElfin »

grafmax wrote: post #94
imiric wrote: post #81

Earlier quoted context omitted.

That's whataboutism at its purest. It's perfectly possible to criticize any government, whether your own or foreign. Claiming that every criticism is tantamount to racism is what's distracting from discussing actual problems.

You’re misunderstanding me. My point is if we were to have sincere solidarity with Chinese people against the international ruling class we would look at our domestic members of that class first. That is simply the practical approach to the problem. The function of the administration’s demonization of China (it’s Sinophobia ) is to 1) distract us from what our rulers have been doing to us domestically and 2) to inspi…

> distract us from what our rulers have been doing to us domestically

America doesn’t have rulers. It has democratically elected politicians. China doesn’t have democracy, however.

> if we were to have sincere solidarity with Chinese people against the international ruling class

There is also no “international ruling class”. In part because there are no international rulers. Speak in more specifics if you want to stick to this claim.

> Concentration camps, genocide, suppressing free speech, suspending due process

I’m not sure what country you are talking about, but America definitely doesn’t fit any of these things that you claim. Obviously there is no free speech in China. And obviously there is no due process if the government can disappear people like Jack Ma for years or punish free expression through social credit scores. And for examples of literal concentration camps or genocide, you can look at Xinjiang or Tibet.

Re: NIST's DeepSeek "evaluation" is a hit piece

#120

Post by edflsafoiewq »

espadrine wrote: post #82
Bengalilol wrote: post #31

Earlier quoted context omitted.

Here's the report: https://www.nist.gov/system/files/documents/2025/09/30/CAISI...

> DeepSeek models cost more to use than comparable U.S. models They compare DeepSeek v3.1 to GPT-5 mini. Those have very different sizes, which makes it a weird choice. I would expect a comparison with GPT-5 High, which would likely have had the opposite finding, given the high cost of GPT-5 High, and relatively similar results. Granted, DeepSeek typically focuses on a single model at a time, instead of OpenAI's appr…

> CAISI chose GPT-5-mini as a comparator for V3.1 because it is in a similar performance class, allowing for a more meaningful comparison of end-to-end expenses.
Post reply on HN