Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Detecting and countering misuse of AI: September 2026

anthropic.com

Post reply on HN
Print view View original

91–100 of 262 posts

Re: Detecting and countering misuse of AI: September 2026

#91

Post by techjamie »

AustinDev wrote: post #73
tedsanders wrote: post #17

Meta: The potential proliferation of biological weapons is serious. Millions could die. It's easy to joke about before it happens, but try to imagine how this thread might look after the successful deployment of a biological weapon by a rogue state or non-state actor. I encourage you to take this topic seriously and contribute posts that add new information or perspectives to the discussion. (I myself think the odds…

How probable do you think it would be that a layman following ChatGPT's bio weapon instructions would just blow themselves up or poison themselves? It's not like positive pressure environments and suits are very common let alone all the other gear you'd need to safely synthesize arbitrary organic molecules. If a rogue state-level actor is doing this then why wouldn't they just kidnap a scientist and hold their family…

All of this also assumes it doesn't hallucinate heavily in the process and give you instructions that are in reality utter nonsense, or create something entirely different from what you were trying to do.

If you've managed to get the equipment and resources to pull this off in the first place, someone with the biological knowledge probably is not the ceiling stopping you from the other part of the problem.

Re: Detecting and countering misuse of AI: September 2026

#92

Post by AustinDev »

14u2c wrote: post #87
AustinDev wrote: post #73

Earlier quoted context omitted.

How probable do you think it would be that a layman following ChatGPT's bio weapon instructions would just blow themselves up or poison themselves? It's not like positive pressure environments and suits are very common let alone all the other gear you'd need to safely synthesize arbitrary organic molecules. If a rogue state-level actor is doing this then why wouldn't they just kidnap a scientist and hold their family…

Why does it have to be a layman? The 2001 anthrax attacks are believe to have been carried out by an insider to the field (senior biodefense researcher). Such tools could potentially be accelerants for similar individuals?

If the five eyes intelligence services aren't monitoring all biodefense researchers and bio chemists around the world all of the time they've probably failed in their jobs.

Re: Detecting and countering misuse of AI: September 2026

#93

Post by kian »

goatlover wrote: post #28
0xWTF wrote: post #23

Earlier quoted context omitted.

The difference between nukes and bioweapons is energy level. Body counts are on the same order of magnitude. Bioweapons are just way easier to make and much harder to detect.

Nukes also destroy infrastructure and irradiate surrounding areas making cleanup and rescue harder. Then there's to potential for massive fires putting debris into the stratosphere, which could dim sunlight for a time and cause global temperatures to fall.

bioweapons tend to not remain contained to the area they're deployed (with notable exceptions for things like Anthrax). Some of them even last a long time on surfaces, making cleanup and rescue harder.

Re: Detecting and countering misuse of AI: September 2026

#94

Post by AustinDev »

techjamie wrote: post #91
AustinDev wrote: post #73

Earlier quoted context omitted.

How probable do you think it would be that a layman following ChatGPT's bio weapon instructions would just blow themselves up or poison themselves? It's not like positive pressure environments and suits are very common let alone all the other gear you'd need to safely synthesize arbitrary organic molecules. If a rogue state-level actor is doing this then why wouldn't they just kidnap a scientist and hold their family…

All of this also assumes it doesn't hallucinate heavily in the process and give you instructions that are in reality utter nonsense, or create something entirely different from what you were trying to do. If you've managed to get the equipment and resources to pull this off in the first place, someone with the biological knowledge probably is not the ceiling stopping you from the other part of the problem.

Yeah, back when this stuff was pretty new in 2024 I found a jailbreak and sent some bio/chem weapon instructions it generated to an organic chem PhD friend. They told me not only were the procedures wrong but that there were several steps that almost certainly would have lead to injury or death. I assume it's gotten better by now but, no way or desire to test.

Re: Detecting and countering misuse of AI: September 2026

#95

Post by palmotea »

sajithdilshan wrote: post #81
tedsanders wrote: post #17

Meta: The potential proliferation of biological weapons is serious. Millions could die. It's easy to joke about before it happens, but try to imagine how this thread might look after the successful deployment of a biological weapon by a rogue state or non-state actor. I encourage you to take this topic seriously and contribute posts that add new information or perspectives to the discussion. (I myself think the odds…

I don't think knowing how to build a bio weapon means one could actually build a bio weapon. It's like saying knowing how to build a nuclear weapon can easily lead to building a nuclear weapon. Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. Also one would need lethal viruses to even start with and…

> Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment.

I think the claim in this report is that it was a state or semi-state actor. They were from "blocked regions" at a "military research institute." So dismissing the threat by solely modeling it as a disgruntled layman rando is not reasonable.

But you totally have to factor in the fact that anything coming out of Anthropic or OpenAI is part of a propaganda campaign to serve their business interests. These threats need to be addressed in ways that cause Anthropic and OpenAI pain (which means damaging their business). Because, FFS, anyone consciously racing to build doomsday devices needs a good, hard slap.

Re: Detecting and countering misuse of AI: September 2026

#96

Post by throwa356262 »

m-hodges wrote: post #88

> We discovered that Moonshot AI, the company that produces the Kimi family of models, silently forwarded customer requests to Claude, instead of processing them using Kimi. Moonshot then displayed Claude’s responses to users. These users thought they were using a Kimi model, but received responses from Claude instead. > DeepSeek also silently relayed exchanges to Claude without informing DeepSeek customers. > MiniMa…

DeepSeek, minimax and so on have razer thin margins but unlike openai and Anthropic they are actually making some profit. Doing this doesn't make any financial sense.

Maybe Anthropic is confusing Chinese AI providers with token resellers using the same alibaba infrastructure? Or maybe something like openrouter was switching between operators depending on price/demand/availability?

Also, how can Anthropic have such accurate information about state actors and cybercriminals? This is the same company that hacked itself and realised that first months later..

Re: Detecting and countering misuse of AI: September 2026

#98

Post by palmotea »

ExoticPearTree wrote: post #29
neom wrote: post #8

Earlier quoted context omitted.

Interesting document. People are vibe coding some crazy shit: "A single Claude subscriber, likely a Bamako-based independent consultant working with Mali’s state intelligence service, the “Agence Nationale de la Sécurité d’État (ANSE),” used Claude to build a system named “Lakana 360,” a population-scale domestic surveillance platform that monitors roughly 25 million SIM cards on all three of the country’s national m…

The Yemeni report is interesting. - How is your missile so accurate? - We're using a vibe coded app on an iPhone that does terrain matching and target finding. The thing is that OK, Anthropic may block it, but nothing says they can't use an open model hosted in a friendly country that has access to GPUs. And yes, this will most likely happen pretty soon to be able to create whatever you want without the AI provider b…

> - We're using a vibe coded app on an iPhone that does terrain matching and target finding.

And that part seems entirely reasonable. It looks like the Tomahawk cruise missile did it with an 84 lb package with a 16-bit computer with 64K of memory [1] [2] and sensors. That's similar computing performance to an original IBM PC, which weighed 30 lb. The only bit of hardware an iPhone doesn't seem to have for TERCOM is a radar altimeter, but it looks like those are available for civil aviation.

[1] https://www.forecastinternational.com/archive/disp_pdf.cfm?D... ("AGM-109/BGM-109 Tomahawk ... Litton 4516-C digital computer with 64K memory")

[2] https://www.forecastinternational.com/archive/disp_old_pdf.c... ("16-bit LC-4516C digital computer")

Re: Detecting and countering misuse of AI: September 2026

#99

Post by Molitor5901 »

Stevvo wrote: post #76

I don't get it. Surely if you were developing novel biological weapons, you would not use a hosted AI service where Anthropic can read what you are doing. And, why would you need to? Any chemistry graduate could make you dozens of highly effective, proven chemical weapons and explosives.

This was my thought. Anyone who has taken secondary biology or chemistry possesses the knowledge, if not at least the ability to find the knowledge, to build all sorts of nasty things. Soil from a farm on slices of potato could yield anthrax spores, this is not specialized or even esoteric knowledge. This news from Anthropic just does not seem as spectacular as I think they might want us to believe, if anything it draws questions around having an AI that cannot be monitored.

Re: Detecting and countering misuse of AI: September 2026

#100

Post by Molitor5901 »

AustinDev wrote: post #13

It was probably just me trying to figure out if I could put one type of draino down the drain within 30 minutes of using a different type. Sorry y'all.

Ah you jest, but your comment reminded me of the person whose home was raided by authorities for researching pressure cookers...

https://www.theguardian.com/world/2013/aug/01/new-york-polic...

Post reply on HN