Building an early warning system for LLM-aided biological threat creation
1–10 of 190 posts
Re: Building an early warning system for LLM-aided biological threat creation
#2Funny company, creates problems for itself to solve.
Re: Building an early warning system for LLM-aided biological threat creation
#3OpenAI seems to be transitioning from an AI lab to an AI fearmongering regulatory mouthpiece.
As someone who lived through the days when encryption technology was highly regulated, I am seeing parallels.
The Open Source cows have left the Proprietary barn. Regulation might slow things. It might even create a new generation of script kiddies and hackers. But you aren't getting the cows back in the barn.
Re: Building an early warning system for LLM-aided biological threat creation
#4Open AI is clearly overestimating the capabilities of its product.
It is kind of funny actually.
Re: Building an early warning system for LLM-aided biological threat creation
#5OpenAI seems to be transitioning from an AI lab to an AI fearmongering regulatory mouthpiece. As someone who lived through the days when encryption technology was highly regulated, I am seeing parallels. The Open Source cows have left the Proprietary barn. Regulation might slow things. It might even create a new generation of script kiddies and hackers. But you aren't getting the cows back in the barn.
Agreed, seems they are sowing FUD by playing on a global disaster event still fresh in short term memory to advance their goal of regulatory capture... the competition isn't letting up so they'd very much like regulation to hamper things
Re: Building an early warning system for LLM-aided biological threat creation
#6Even full-strength GPT-4 can spout nonsense when asked to come up with synthetic routes for chemicals. I am skeptical that it's more useful (dangerous) as an assistant to mad scientist biologists than to mad scientist chemists.
For example, from "Prompt engineering of GPT-4 for chemical research: what can/cannot be done" [1]
GPT-4 also failed to solve application problems of organic synthesis. For example, when asked about a method to synthesize TEMPO, it returned a chemically incorrect answer (Scheme 2, Prompt S 8). The proposal to use acetone and ammonia as raw materials was the same as the general synthesis scheme of TEMPO. However, it misunderstood the aldol condensation occurring under primary conditions in this process as an acid-catalyzed reaction. Furthermore, it asserts that 2,2,6,6-tetramethylpiperidine (TMP) is produced by an inadequately explained "reduction process." In reality, after promoting the aldol condensation further to generate 4-oxo-TMP, TMP is produced by reduction with hydrazine and elimination under KOH conditions. GPT-4 may have omitted this series of processes.
The scheme after obtaining TMP was also chemically inappropriate. Typically, TEMPO can be obtained by one-electron oxidation of TMP in the presence of a tungsten catalyst and H2O2. However, GPT-4 advocated the necessity of excessive oxidation reactions: the formation of oxoammonium by H2O2 oxidation in the presence of hydrochloric acid, and further oxidation with sodium hypochlorite. Two-electron oxidation is already performed in the first oxidation stage, which goes beyond the target product. There is no chemical meaning to adding NaClO in that state. This mistake probably occurred due to confusion with the alcohol oxidation reaction by TEMPO (requiring an oxidizing agent under acidic conditions).
And this is for a common compound that would have substantial representation in the training data, rather than a rare or novel molecule.
[1] https://chemrxiv.org/engage/api-gateway/chemrxiv/assets/orp/...
Re: Building an early warning system for LLM-aided biological threat creation
#7OpenAI seems to be transitioning from an AI lab to an AI fearmongering regulatory mouthpiece. As someone who lived through the days when encryption technology was highly regulated, I am seeing parallels. The Open Source cows have left the Proprietary barn. Regulation might slow things. It might even create a new generation of script kiddies and hackers. But you aren't getting the cows back in the barn.
"However, the obtained effect sizes were not large enough to be statistically significant, and our study highlighted the need for more research around what performance thresholds indicate a meaningful increase in risk."
"We also discuss the limitations of statistical significance as an effective method of measuring model risk"
Seriously?
Re: Building an early warning system for LLM-aided biological threat creation
#8I bet the improvements (small as they are) are mostly in filling out paperwork and drafting emails
Re: Building an early warning system for LLM-aided biological threat creation
#9So, the model is bad at helping in this particular task.
How does this compare with a control of a beneficial human task? Like someone in a lab testing blood samples or working on cancer research?
Is the model equally useless for those types of lab tasks?
What about other complex tasks, like home repair or architecture?
Is this a success of guardrails or a failing of the model in general?
Re: Building an early warning system for LLM-aided biological threat creation
#10Open AI is clearly overestimating the capabilities of its product. It is kind of funny actually.
I also think it’s ludicrous to the point of hilarity; but it’s also harmful as people who can make laws and big decisions are buying this horse shit.