>These are things you can just look up. The LLM contributes nothing that the internet didn't already provide
The internet doesn't allow one to design _new_ weapons, possibly way more effective (which if you read carefully the first story, this one does).
>I'm not sure that LLMs significantly increase the attack surface here.
Being able to ask an AI to develop new deadly varieties which we'll not be able to detect or cure, and may be easy to produce, doesn't increase attack surface?
>Also, sarin gas doesn't pose an existential risk to humanity
Is x-risk the only thing we care about? The entire thread started with arguing x-risk is a distraction. I would be very slightly more comfortable with that argument if people took 'ordinary' risks seriously.
As it is, all camps have their heads in the sand in different ways. The illusion here is that AI advancement changes nothing, so the only thing worth discussing are variations of the current culture war issues, when obviously it does change everything even if completely put aside AGI/alignment arguments. e.g. If labour won't matter for productivity that has very grim political implications.
>To my point, they didn't "try"!!! They DID.
That's the point. Give me an x-risk scenario the doomers warn about, and I'll find you a group of humans which very much want the exact scenario (or something essentially indistinguishable for 99% of humanity) to happen and will happily use AI if it helps them. Amusingly, alignment research is unlikely to help there - it can be argued to increase the risk from humans.
>there are some mitigations we could introduce such that the barrier to using LLMs for this sort of thing
There are many things we can do in theory to mitigate all sorts of issues, which have the nice property of never ever being done. e.g. Yud's favorite disaster scenario appears to be a custom built virus. This relies on biolabs accepting random orders, which leaves the question of why are we allowing this at all (AI or not)? There's no good reason for allowing most crypto to exist, given its current effects on society, even before AGI comes into question, yet we allow it for what reason exactly?
If there's any risk here at all, we can safely rely on humanity doing nothing before anything happens - but in the case of x-risk actually existing, there's no reason to assume we'll have a second chance.