Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NIST's DeepSeek "evaluation" is a hit piece

erichartford.com

Post reply on HN
Print view View original

161–170 of 251 posts

Re: NIST's DeepSeek "evaluation" is a hit piece

#161

Post by xpe »

garyfirestorm wrote: post #153
xpe wrote: post #38

Earlier quoted context omitted.

Of course there will be some degree of governmental and/or political influence. The question is not if but where and to what extent. No one should proclaim "bullshit" and wave off this entire report as "biased" or useless. That would be insipid. We live in a complex world where we have to filter and analyze information.

did you even read the article? when you download open source deep seek model and run it yourself - zero packets are being transmitted. thereby disproving the fundamental claim in the NIST report (additionally NIST doesn't provide any evidence to support their claim) This is basic science and no amount of politicking should ever challenge something this fundamental!

> when you download open source deep seek model and run it yourself - zero packets are being transmitted. thereby disproving the fundamental claim in the NIST report (additionally NIST doesn't provide any evidence to support their claim)

You are confused about what the NIST report claimed. Please review the NIST report and try to find a quote that matches up with what you just said. I predict you won’t find it. Prove me wrong?

Please review the claims that the NIST report actually makes. Compare this against Eric Hartford’s article. When I do this, Hartford comes across as confused and/or intellectually dishonest.

Re: NIST's DeepSeek "evaluation" is a hit piece

#162

Post by xpe »

garyfirestorm wrote: post #153
xpe wrote: post #38

Earlier quoted context omitted.

Of course there will be some degree of governmental and/or political influence. The question is not if but where and to what extent. No one should proclaim "bullshit" and wave off this entire report as "biased" or useless. That would be insipid. We live in a complex world where we have to filter and analyze information.

did you even read the article? when you download open source deep seek model and run it yourself - zero packets are being transmitted. thereby disproving the fundamental claim in the NIST report (additionally NIST doesn't provide any evidence to support their claim) This is basic science and no amount of politicking should ever challenge something this fundamental!

> did you even read the article?

I am not going to dignify this with a response.

Please review the hacker news guidelines.

Re: NIST's DeepSeek "evaluation" is a hit piece

#163

Post by nylonstrung »

AnthonyMouse wrote: post #112
riehwvfbk wrote: post #40

Earlier quoted context omitted.

So in other words, they can make their LLM disagree with the preferred narrative of the current US administration? Inconceivable! Note that the value of $current_administration changes over time. For some reason though it is currently fashionable in tech circles to disagree with it about ICE and H1B visas. Maybe it's the CCP's doing?

It's not about the current administration. They can, for example, train it to emit criticism of democratic governance in favor of state authoritarianism or omit valid counterarguments against concentrating world-wide manufacturing in China.

Deepseek IME is wildly less censored than the western closed weights models unless you want to ask about Tiananmen Square to prove a point

The political benchmarks show it's political slant is essentially identical to the other models, all of which place in the "left libertarian" quadrant of the political compass

Re: NIST's DeepSeek "evaluation" is a hit piece

#164

Post by nylonstrung »

AnthonyMouse wrote: post #107
BowBun wrote: post #2

I'm not at all surprised, US agencies have long since been political tools whenever the subject matter crosses national borders. I appreciate this take as someone who has been skeptical of Chinese electronics. While I agree this report is BS and xenophobic, I am still willing to bet that either now or later, the Chinese will attempt some kind of subterfuge via LLMs if they have enough control. Just like the US would,…

> While I agree this report is BS and xenophobic, I am still willing to bet that either now or later, the Chinese will attempt some kind of subterfuge via LLMs if they have enough control. The answer to this isn't to lie about the foreign ones, it's to recognize that people want open source models and publish domestic ones of the highest quality so that people use those.

Lol remember when Perplexity made a "1776" version of Deepseek with half the benchmarks that still wouldn't answer the "censored" questions about the CCP

Re: NIST's DeepSeek "evaluation" is a hit piece

#165

Post by nylonstrung »

wordpad wrote: post #55
Bengalilol wrote: post #31

Earlier quoted context omitted.

Here's the report: https://www.nist.gov/system/files/documents/2025/09/30/CAISI...

TLDR for others: * DeepSeek cutting edge models are still far behind * On par DeepSeek costs 35% more to run * DeepSeek models 12 times more susceptible to jail breaking and malicious instructions * DeepSeek models follow strict censorship I guess none of these are a big deal to non-enterprise consumers.

Saying Deepseek is more expensive is FUD

Token price on 3.2 exp is <5% what the US LLMs are and it's very close in benchmarks. Which we know that ChatGPT, Google, Grok and Claude have explicitly gamed to inflate their capabilities

Re: NIST's DeepSeek "evaluation" is a hit piece

#166

Post by xpe »

Take away #1: Eric Hartford’s article is deeply confused. (I’ve made many other specific comments that support this conclusion.)

Take away #2: as evidenced by many comments here, many HN commenters have failed to check the source material themselves. This has led to a parade of errors.

I’m not here to say that I’m better than that because I’ve screwed up a’plenty. We all make mistakes sometimes. We can choose to recognize and learn from them.

I am saying this: as a community we can and should aim higher. We can start by owning our mistakes.

Re: NIST's DeepSeek "evaluation" is a hit piece

#167

Post by nylonstrung »

getdoneist wrote: post #5

Let them demonize it. I'll use the capable and cheap model and gain competitive advantage.

Yeah it's absurd how people will defend closed source, even more censored models that cost >20x more for equivalent quality and worse speed

The Chinese companies aren't benchmark obsessed like the western Big Tech ones and qualitatively I feel Kimi, GLM and Deepseek blow them away even though on paper they benchmark worse in English

Kimi gives insanely detailed answers on hardware questions where Gemini and Claude just hallucinate, probably because it uses Chinese training data better

Re: NIST's DeepSeek "evaluation" is a hit piece

#168

Post by FooBarWidget »

SilverElfin wrote: post #118
FooBarWidget wrote: post #46

Earlier quoted context omitted.

They revoke passports of personnel whom they deem are at risk of being negatively influenced or even kidnapped when abroad. Re influence, think school teachers. Re kidnapping, see Meng Wangzhou (Huawei CFO). There is a history of important Chinese personnel being kidnapped by e.g. the US when abroad. There is also a lot of talk in western countries about "banning Chinese [all presumed spies/propagandists/agents] from…

You’re twisting the (obvious) truth. These people are being held prisoners because they’re of economic value to the party. And they would probably accept a job and life elsewhere if they weee given enough money. They are not being held prisoners for their own protection.

Let's just say "things can happen for reasons other than 'the govt is evil'" is not only an opinion that exists, but is also valid. You seem to be severely underestimating just how much of a 9/11 moment Meng Wanzhou is for Chinese. You should talk to more mainlanders. This isn't 20 years ago anymore.

Re: NIST's DeepSeek "evaluation" is a hit piece

#169

Post by ACCount37 »

nylonstrung wrote: post #165
wordpad wrote: post #55

Earlier quoted context omitted.

TLDR for others: * DeepSeek cutting edge models are still far behind * On par DeepSeek costs 35% more to run * DeepSeek models 12 times more susceptible to jail breaking and malicious instructions * DeepSeek models follow strict censorship I guess none of these are a big deal to non-enterprise consumers.

Saying Deepseek is more expensive is FUD Token price on 3.2 exp is <5% what the US LLMs are and it's very close in benchmarks. Which we know that ChatGPT, Google, Grok and Claude have explicitly gamed to inflate their capabilities

And we "know" that how, exactly?

Re: NIST's DeepSeek "evaluation" is a hit piece

#170

Post by nylonstrung »

dns_snek wrote: post #14
Hizonner wrote: post #10

Earlier quoted context omitted.

> I am still willing to bet that either now or later, the Chinese will attempt some kind of subterfuge via LLMs if they have enough control. Like what, exactly?

Like generating vulnerable code given a specific prompt/context. I also don't think it's just China, the US will absolutely order American providers to do the same. It's a perfect access point for installing backdoors into foreign systems.

Why would they do this? Deepseek is a private company not owned by the CCP.

There's zero reason or even technical feasibility for them to skip in backdoor that would be easily detected and destroy their market share

None of the security benchmarks or audits show that any Chinese models write insecure code

Post reply on HN