> Balancing chemical equations, Arrhenius acids and bases, basic lab work, and ...
... oxidation, exothermic reactions?
A teacher who never actually studied the subject and only has knowledge of the pre-filtered material from the textbook would be little more than a talking textbook. I mean, textbooks are useful, but we already have those.
> Without a recipe or off-the-shelf explosives?
If they've ever studied the relevant subjects, yes.
> And even if true, a teacher who hasn't put themselves to the task of building a pipe bomb can be a perfectly good chemistry teacher.
But the suggestion to pre-filter is more like a teacher who was never allowed to learn enough to succeed at the task even if they tried. Teachers who could succeed but refuse to try are the norm. (Or maybe they have tried on their own time, or done the thought experiment, but they know that handing out step-by-step instructions would be taboo in most circumstances.)
> These models can't figure anything out, they just regurgitate and associate on command.
I think you may be overemphasizing "regurgitate" and underemphasizing "associate". ChatGPT can emit instructions for just about anything that were clearly not just regurgitated from some other source, and surprisingly often they're even accurate - or accurate enough to use as a starting point.
Let's just agree that pre-filtering the training text is not a practical way to prevent these models from producing emissions that are considered dangerous or taboo, at least not without losing the traits that make them so interesting in the first place.
But the self-censoring hidden prompts that they're using can also lead to absurd results, so there's a lot of room for improvement. I was trying to generate a fictional story where at one point the main character tries walk out of a store before paying for an item, and it refused to write something "illegal or unethical". It was okay with having the character leave some money on the counter before walking out.