Ask HN: AI Panic, you buying it?

Yes, they are.

https://www.anthropic.com/threat-intelligence-report-septemb...

> Biological misuse is one of the most serious risks of frontier AI models. It has long been a concern that AI models might one day reach the level of capability where they can help to make existing pathogens more dangerous—or create entirely new ones. Without the correct safeguards, such capabilities could have catastrophic consequences.

Results from evaluations of older models (for example Claude Opus 4 and Claude Sonnet 4.5, from 2025) clearly showed that these models were well below the threshold where they could meaningfully assist a sophisticated user in carrying out dangerous biological research. As a result, safeguards on these models were less stringent, directed mostly at preventing access to content that might uplift novices in recreating known bioweapons. But for today’s models—which are capable of assisting in a range of complex scientific research tasks—the evidence is no longer certain, and we cannot make that same assurance. For this reason, and out of an abundance of caution, we have launched recent models (most notably Claude Fable 5) with stronger safeguards that restrict access to a wide range of dual-use biological research queries.

...

Here, we present five case studies of actors using our models in ways that could support biological weapons development. These examples are illustrative of the kinds of tasks to which our models are put, and the often-difficult judgements we have to make when assessing whether or not a given biological use is dangerous. They also convey that we encounter what would otherwise be non-public insight into the risks associated with biological misuse from AI

Astra, Claude -- these aren't "Jesus on a keyboard" somewhere. They don't assemble into one big brain and then answer everyone at the speed of light.

It's weights. Spun up billions of times over.

If humans are taking the outputs as scripture? As "intelligence we must obey"? Yeah, we're in for a bad time. "Intelligence we want running core infrastructure?" Puke.

But why would we do that? Why would we hook up an LLM to a robot factory, or make an ASI that could easily move itself into the ether?

Everything magic about LLMs is what harness we put them in and let them drive. There's nothing dangerous about predicting the next token. There's something dangerous about letting them do dangerous things, but nobody seems to care for maybe Tesla FSD. We don't think it's about to take over.

Any assignment of motive or whatever to the AI, you're talking about one cobbled-together harness or something, with an LLM manning the controls. You can swap the brain out from Claude to Astra and the motivated-AI will maintain its motivation--it lives outside of the inference or the data center. So yeah, let's not let it take control of the nukes. That would be really dangerous. But it's not going to take control on its own. "It" is a calculator.

Anthropic obviously has guardrails to prevent people from using AI to do all kinds of things already. All I hear about is you guys bitching it won't do your barely-related-to-security work. That's all we need to continue doing to have AI safety. Stop humans from doing evil and stupid things with it. The idea that the datacenter is going to assemble its own sentience and motivations is pure retardium (at this time, for llms).

What is the concrete scenario that is based on reality which results in the percentage increasing above zero?

LLMs don't need to be the dangerous ASI thing themselves for them to still be dangerous. It suffices if they

- destabilize the world just by what they do: making previously very hard things (generating fake everything, manipulating people, hacking, bio-weapons research) easy enough fast enough that our civilization doesn't cope with it and/or

- accelerate AI research to the point where they make the dangerous ASI thing appear years earlier than would otherwise have happened

I think there are undeniable and novel risks with AI. But there are risks in everything. We could theorize all day about them. We all use tools every day that could be used for very ugly acts. An iPhone can be used to plan a terrorist attack. A 3D printer can be used to print weapons. We could sit here and fear monger all day. Time will be needed to understand, weigh and contextualize these risks.

What is a more immediate concern for me is the efforts of frontier labs to have government craft legislation that shields them from competition, cements them as monopolies, or assigns them the capability to be moral arbiters in our society.