Given it had write access to its implementation, "unexpected" here reads more like "inevitable" than the witchcraft the article wishes to imply.
I think the salient point is that it was unexpected to the researchers. When people propose regulation for AI systems out of concern for unexpected self-improvement, this is the sort of scenario they are worried about. Security in general is hard to get right, but the basics are not difficult if you try, and most companies fail to even implement the basics. For AI, sandboxing will get you quite far, but how do you gu…
We have memes, which are eye worms, who have adapted like cats to co-exist with us, informationally, and like cats, they wish for our demise.
Methinks that sand boxing is more like cat boxing, vs say an carrier.