Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NIST's DeepSeek "evaluation" is a hit piece

erichartford.com

Post reply on HN
Print view View original

41–50 of 251 posts

Re: NIST's DeepSeek "evaluation" is a hit piece

#41

Post by im3w1l »

Hizonner wrote: post #10
BowBun wrote: post #2

I'm not at all surprised, US agencies have long since been political tools whenever the subject matter crosses national borders. I appreciate this take as someone who has been skeptical of Chinese electronics. While I agree this report is BS and xenophobic, I am still willing to bet that either now or later, the Chinese will attempt some kind of subterfuge via LLMs if they have enough control. Just like the US would,…

> I am still willing to bet that either now or later, the Chinese will attempt some kind of subterfuge via LLMs if they have enough control. Like what, exactly?

You make it say that China is good, Chinese history is good, West is bad, western history is bad. Republicans are bad and democrats are bad too and so are Europe parties. If someone asks for how to address issues in their own life it references Confucianism and modern Chinese thinkers and communist party orthodoxy. If someone wants to buy a product you recommend a Chinese one.

This can be done subtly or blatantly.

Re: NIST's DeepSeek "evaluation" is a hit piece

#42

Post by Hizonner »

dns_snek wrote: post #14
Hizonner wrote: post #10

Earlier quoted context omitted.

> I am still willing to bet that either now or later, the Chinese will attempt some kind of subterfuge via LLMs if they have enough control. Like what, exactly?

Like generating vulnerable code given a specific prompt/context. I also don't think it's just China, the US will absolutely order American providers to do the same. It's a perfect access point for installing backdoors into foreign systems.

Up until recently, I would have reminded you that the US government (admittedly unlike the Chinese government) has no legal authority to order anybody to do anything like that. Not only that, but if it asked, it'd be well advised to ask nicely, because it also has no legal authority to demand that anybody keep such a request secret. And no, evil as it is, the "National Security Letter" power doesn't in fact cover anything like that.

Now I'm not sure legality is on-topic any more.

Re: NIST's DeepSeek "evaluation" is a hit piece

#43

Post by edflsafoiewq »

evv wrote: post #34
xpe wrote: post #17

Please don't just read Eric Hartford's piece. Start with the key findings from the source material: "CAISI Evaluation of DeepSeek AI Models Finds Shortcomings and Risks" [1]. Here are the single-sentence summaries: DeepSeek performance lags behind the best U.S. reference models. DeepSeek models cost more to use than comparable U.S. models. DeepSeek models are far more susceptible to jailbreaking attacks than U.S. mod…

It's funny how they mixed in proprietary models like GPT-5 and Anthropic with the "comparable U.S. models". Until they compare open-weight models, NIST is attempting a comparison between apples and airplanes.

They compare with gpt-oss.

Re: NIST's DeepSeek "evaluation" is a hit piece

#44

Post by CamperBob2 »

Bengalilol wrote: post #39
CamperBob2 wrote: post #21

Earlier quoted context omitted.

Hardly the same thing. Ask Gemini or OpenAI's models what happened on January 6, and they'll tell you. Ask DeepSeek what happened at Tiananmen Square and it won't, at least not without a lot of prompt hacking.

Ask Grok to generate an image of bald Zelensky: it does execute. Ask Grok to generate an image of bald Trump: it goes on with an ocean of excuses on why the task is too hard.

I don't use Grok. Grok answers to someone with his own political biases and motives, many of which I personally disagree with.

And that's OK, because nobody in the government forced him to set it up that way.

Re: NIST's DeepSeek "evaluation" is a hit piece

#46

Post by FooBarWidget »

falcor84 wrote: post #27
JPKab wrote: post #23

The CCP literally revoked the visas of key DeepSeek engineers. That's all we need to know.

I would like to know more

They revoke passports of personnel whom they deem are at risk of being negatively influenced or even kidnapped when abroad. Re influence, think school teachers. Re kidnapping, see Meng Wangzhou (Huawei CFO).

There is a history of important Chinese personnel being kidnapped by e.g. the US when abroad. There is also a lot of talk in western countries about "banning Chinese [all presumed spies/propagandists/agents] from entering". On a good faith basis, one would think China banning people from leaving is a good thing that aligns with western desires, and should thus be applauded. So painting the policy as sinister tells me that the real desire is something entirely different.

Re: NIST's DeepSeek "evaluation" is a hit piece

#47

Post by OrvalWintermute »

Since a major part of the article covers cost expenditures, I am going to go there.

I don't think it is possible to trust DeepSeek as they haven't been honest.

DeepSeek claimed "their total training costs amounted to just $5.576 million"

SemiAnalysis "Our analysis shows that the total server CapEx for DeepSeek is ~$1.6B, with a considerable cost of $944M associated with operating such clusters. Similarly, all AI Labs and Hyperscalers have many more GPUs for various tasks including research and training then they they commit to an individual training run due to centralization of resources being a challenge. X.AI is unique as an AI lab with all their GPUs in 1 location."

SemiAnalysis "We believe the pre-training number is nowhere the actual amount spent on the model. We are confident their hardware spend is well higher than $500M over the company history. To develop new architecture innovations, during the model development, there is a considerable spend on testing new ideas, new architecture ideas, and ablations. Multi-Head Latent Attention, a key innovation of DeepSeek, took several months to develop and cost a whole team of manhours and GPU hours.

The $6M cost in the paper is attributed to just the GPU cost of the pre-training run, which is only a portion of the total cost of the model. Excluded are important pieces of the puzzle like R&D and TCO of the hardware itself. For reference, Claude 3.5 Sonnet cost $10s of millions to train, and if that was the total cost Anthropic needed, then they would not raise billions from Google and tens of billions from Amazon. It’s because they have to experiment, come up with new architectures, gather and clean data, pay employees, and much more."

Source: https://semianalysis.com/2025/01/31/deepseek-debates/

Re: NIST's DeepSeek "evaluation" is a hit piece

#49

Post by xpe »

BowBun wrote: post #2

I'm not at all surprised, US agencies have long since been political tools whenever the subject matter crosses national borders. I appreciate this take as someone who has been skeptical of Chinese electronics. While I agree this report is BS and xenophobic, I am still willing to bet that either now or later, the Chinese will attempt some kind of subterfuge via LLMs if they have enough control. Just like the US would,…

> While I agree this report is BS and xenophobic

Examples please? Can you please share where you see BS and/or xenophobia in the original report?

Or are you basing your take only on Hartford's analysis? But not even Hartford make any claims of "BS" or xenophobia.

It is common throughout history for a nation-state to worry about military and economic competitiveness. Doing so isn't necessarily isn't necessarily xenophobic.

Here is how I think of xenophobia, as quoted from Claude (which to be honest, explains it better than Wikipedia or Brittanica, in my opinion): "Xenophobia is fundamentally about irrational fear or hatred of people based on their foreign origin or ethnicity. It targets people and operates through stereotypes, dehumanization, and often cultural or racial prejudice."

According to this definition, there is zero xenophobia in the NIST report. (If you disagree, point to an example and show me.) The NIST report, of course, implicitly promotes ideals of western democratic rule over communist values -- but to be clear, this isn't xenophobia at work.

What definition of xenophobia are you using? We don't have to use the same exact definition, but you should at least explain yours if you want people to track.

Post reply on HN