Live data from Hacker News

Detecting and countering misuse of AI: September 2026

anthropic.com

211–220 of 262 posts

Re: Detecting and countering misuse of AI: September 2026

#211

Meta: The potential proliferation of biological weapons is serious. Millions could die. It's easy to joke about before it happens, but try to imagine how this thread might look after the successful deployment of a biological weapon by a rogue state or non-state actor. I encourage you to take this topic seriously and contribute posts that add new information or perspectives to the discussion. (I myself think the odds…

I don't think knowing how to build a bio weapon means one could actually build a bio weapon. It's like saying knowing how to build a nuclear weapon can easily lead to building a nuclear weapon. Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. Also one would need lethal viruses to even start with and…

> Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. Also one would need lethal viruses to even start with and that's not something you can order off of amazon.

This is laughably misinformed. You can in fact build a bio weapon in a glorified shed if you know what you're doing. However it will be quite involved, requiring experience on the bench and a great deal of attention to small details. In short an LLM can't suddenly morph you into a molecular biology lab tech with 5+ years of experience.

Meanwhile as with any STEM discipline the educational process effectively serves as a screen for being a reasonably well adjusted adult.

> Further, even to build a chemical weapon, the compounds needed are strictly controlled almost in every country

You can synthesize from basic precursors but you will hit the same issue as above. You will need actual experience on the bench and the process of getting that is going to screen out the vast majority of would be bad actors. (Notably it failed to screen out the members of Aum Shinrikyo but that is very much the exception.)

https://en.wikipedia.org/wiki/Aum_Shinrikyo

Re: Detecting and countering misuse of AI: September 2026

#212

Earlier quoted context omitted.

All of this also assumes it doesn't hallucinate heavily in the process and give you instructions that are in reality utter nonsense, or create something entirely different from what you were trying to do. If you've managed to get the equipment and resources to pull this off in the first place, someone with the biological knowledge probably is not the ceiling stopping you from the other part of the problem.

Yeah, back when this stuff was pretty new in 2024 I found a jailbreak and sent some bio/chem weapon instructions it generated to an organic chem PhD friend. They told me not only were the procedures wrong but that there were several steps that almost certainly would have lead to injury or death. I assume it's gotten better by now but, no way or desire to test.

TBF an inexperienced person following _correct_ directions that involve anything even remotely dangerous is also exceedingly likely to injure or kill themselves.

Re: Detecting and countering misuse of AI: September 2026

#213
> Our investigation revealed that DeepSeek also deployed tactics similar to Moonshot’s. DeepSeek built a CoT extraction pipeline, relying on the same cross-session replay attack described above. DeepSeek also silently relayed exchanges to Claude without informing DeepSeek customers. Like GTG-16002, their customers were likely not made aware that their requests were being funneled to Claude.

If true, would that sort of explain why Chinese Models score high on benchmarks, but not quite as capable when given real tasks?

Re: Detecting and countering misuse of AI: September 2026

#214

Quite the double standard here... Conventional Weapons -We identified a cell of threat actors based in northern Yemen -We identified a China-based threat actor who used Claude -We identified likely freelance Russia-based threat actors -We identified a China-based actor who used Claude’s chat -In this case, a Russia-based actor used Claude -We identified a China-based threat actor who used Claude Biological misuse We…

How is this a double standard? A cell of actors in northern Yemen building guided rockets was not working on a PhD dissertation. You are allowed to use common sense sometimes.

Israel, probably

Re: Detecting and countering misuse of AI: September 2026

#217

> We discovered that Moonshot AI, the company that produces the Kimi family of models, silently forwarded customer requests to Claude, instead of processing them using Kimi. Moonshot then displayed Claude’s responses to users. These users thought they were using a Kimi model, but received responses from Claude instead. > DeepSeek also silently relayed exchanges to Claude without informing DeepSeek customers. > MiniMa…

DeepSeek, minimax and so on have razer thin margins but unlike openai and Anthropic they are actually making some profit. Doing this doesn't make any financial sense. Maybe Anthropic is confusing Chinese AI providers with token resellers using the same alibaba infrastructure? Or maybe something like openrouter was switching between operators depending on price/demand/availability? Also, how can Anthropic have such ac…

> Maybe Anthropic is confusing Chinese AI providers with token resellers using the same alibaba infrastructure? Or maybe something like openrouter was switching between operators depending on price/demand/availability?

Or maybe Anthropic is scared shitless of those competitors and is trying anything to smear them.

Don't forget their goal is to ban open source and foreign AI. Being the sole legal provider is their business plan.

Re: Detecting and countering misuse of AI: September 2026

#218

> We discovered that Moonshot AI, the company that produces the Kimi family of models, silently forwarded customer requests to Claude, instead of processing them using Kimi. Moonshot then displayed Claude’s responses to users. These users thought they were using a Kimi model, but received responses from Claude instead. > DeepSeek also silently relayed exchanges to Claude without informing DeepSeek customers. > MiniMa…

Consider me incredibly skeptical of any of these claims.

Same, I don't even see how that would work since you see the full thinking traces in Kimi but are hidden with Claude.

And the Deepseek one sounds even more dubious as Deepseek is one of the cheapest model around, why relay anything to a more expensive model? I'm sure even the gray market Claude prices are still higher than Deepseek.

Re: Detecting and countering misuse of AI: September 2026

#219

Earlier quoted context omitted.

Meta: These unregulated companies can't/won't properly prevent their technology from committing felonies against other companies. Why should we trust them to control bioweapon development without regulations? They themselves are non-state actors.

The question isn't only "should we" regulate the companies, but also "how"? The current Trump Admin can't even decide on the first question, let alone come up for a plan on the second. The Trump Admin's EO was to ask nicely all of the AI companies to give them 30 days to voluntarily review each model before wide release, but they have also failed to do that for the Mythos/Fable release, only to get a call from Amazon…

[deleted]

Re: Detecting and countering misuse of AI: September 2026

#220
post #197

They seem to be mixing together things that are actually harmful to the public, with things that are merely harmful to their business model (which is their claim that they can grab whatever data that they want regardless of the wishes of the owners of the data and use it to improve their models, but competitors can't do that to them).

Universalising one’s personal experience and needs is a childish trait most people grow out of.

Ofcourse you’ll still see companies, governments, C-suites justifying their own personal needs with “we need X Y Z”.

Post reply on HN