This experiment has been run, and even the biological stuff is far fetched.
I've written about this before, but the issue is that these people have worked with computers their entire lives, and they keep projecting outwards from there.
The 20th century was dominated by mad scientists (mostly Teller) and generals (bombs away LeMay), but one thing that I respect about them is that they didn't just sit and guess about probable futures. They sat down and actually ran experiments.
For example, in the 1960s, there was a whole generation of people asking "who's next?" after the US, USSR, the UK and France had made The Bomb. And instead of just gesticulating wildly and trying to keep fighting the losing fight of "classify everything, admit nothing," Teller found three smart, young physicists (postdocs) with 0 nuclear weapons experience, and indeed 0 weapons experience, and ran an experiment called the Nth Country Experiment, where they were were asked to make a design for a working bomb.
After 2+ years, they were able to. And so proliferation work shifted from controlling knowledge about the technology and the technology itself to materials.
Something similar was done with bioweapons via Project Bacchus, where they gave actual bioweapons experts carte blanche to set up a secret bioweapons lab with COTS equipment. They succeeded. This was in the early 2000s.
This then led to surveillance of specific purchase combinations and equipment. Because tbh, most weapons of mass destruction are commodity technology. Nuclear weapons are 80+ years old. Chemical weapons are over a century old. Bioweapons are god knows how much older. And it's not the knowledge of these things that matter, but intent, materials, and the ability to create them.
Just because you know something about a thing doesn't mean that you're capable of doing that thing.
Let's take bioweapons.
They keep comparing wet work in a lab to writing code on a computer.
When you screw up an exploit, you fail to execute the exploit. Famously, just like software's near zero marginal cost of distribution, the marginal cost of failure is nearly zero.
You can screw up an infinite number of times on your way to a successful exploit.
If you screw up with lethal agents in a lab? You die.
Here's a non-exhaustive list,
Dora Lush died after accidentally pricking her finger with a needle containing lethal scrub typhus while attempting to develop a vaccine for the disease
A 23-year-old laboratory assistant at the London School of Hygiene and Tropical Medicine, was infected with smallpox after observing the harvesting of live smallpox virus from eggs without isolation cabinets at that time. The assistant was hospitalised and before being isolated, she infected two visitors to a patient in an adjacent bed, both of whom died. They in turn infected a nurse, who survived
Ebola laboratory infection by the accidental stick of contaminated needle in the United Kingdom
Researcher Nikolai Ustinov was lethally infected with the Marburg virus after accidentally pricking himself with a syringe used for inoculation of guinea pigs. The accident occurred at the Scientific-Production Association "Vektor" (today the State Research Center of Virology and Biotechnology "Vektor") in Koltsovo, USSR (today Russia).
"lethally infected with the Marburg virus after accidentally pricking himself"
Anything lethal enough to kill other humans is lethal enough to kill you.
And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.
And this, of course, doesn't even cover the cost of equipment, the precursors, sourcing the highly specific materials needed, then setting the equipment up... etc.
The same is true for the Bosch-Haber / Haber-Bosch process, which famously made WW1 possible. Every HS'er learns about the process and the steps. Steps that were classified once upon a time and were the subject of negotiation at the Versailles.
Does that mean a HS'er (or any adult) can set up an experiment that works at 177 times the pressure of the Earth's atmosphere to do anything at any scale without significant infrastructure and help?
No, it doesn't work like that.
The people who can do this are domain experts, and they've been able to do this with COTS stuff since the 1990s, at the very least, for a price of around $2M. And those people don't need a LLM to tell them what to do. In fact, they're the exact people who'll have access to unrestricted versions of these LLMs.
And from a security perspective, I would bet good money that flooding the FBI's tip line with junk about every teenager trying to learn "what be a mitochondria" does more harm to the effort of finding people who could be planning such a thing than it helps. It takes more resources to go through the mass of false negatives that have now been created as matter of policy.
These experiments have been run. And we can run them again.
The fictional scenario of someone learning how bioweapons work and conjuring up a plague isn't real and it hurts humanity as a whole to impede the sciences over it.
Because what someone can flail around in / do is learn about immunology / try to "cure cancer" with a LLM and hopefully get started on a long career in medicine. Or, a discovery that matters.
Because in those cases, if and when they do end up at a lab, screwing up doesn't mean death. Just tons of wasted time (and money). And they will fail / screw up. Just look at literally every undergrad in any lab and the expensive messes they create.