Loved the Asimov robotics stories as a kid. Many fond memories of Susan Calvin's investigations. Yishan Wong would like us to generalize from this event as follows:
> It demonstrates quite conclusively that with all our current alignment work, that even at the level of our current LLMs, we are absolutely terrible at predicting how it’s going to execute an intended set of instructions.
But this misses the mark. It doesn't demonstrate that. Does Yishan Wong think these examples were found by accident or brute force search? It was trivial to predict how Gemini would behave for anyone who knows anything about the modern Google culture, which is why so many adversarial examples appeared immediately, the moment people outside of Google's management chain were able to get access.
This isn't the first time such things have happened. Prior generations of ML model have experienced analogous situations. I used to work on the Gmail team trying to stop people sending spam from Gmail accounts. Many moons ago we had an awkward situation: the enterprise Gmail product managers had been implicitly assuming that spammers wouldn't pay for the ability to send spam, and therefore if you had a verified credit card on file you were legit and should have high sending quotas. Well that doesn't follow at all, and so the spam ratio for commercial users had been steadily climbing to be much higher than for consumer accounts. The outbound spam filter had an ML model on it that was continuously training, and one day it concluded that being a commercial account was such a strong indicator of badness it should just block all emails sent by all paying customers. Outage time! The model was quickly disabled, as fortunately most spam was filtered by deterministic human written logic so the loss wasn't too bad. But then we faced two problems:
1. How to bring it back?
2. How to explain what happened to upper management?
We attempted to report that the root cause was negligence by another part of the company (who had been warned of the growing problem in advance, many times). Needless to say this type of speaking truth to power is not easy and didn't work. No company wants to hear that a lot of their apparent customers are illegitimate and should be booted, they have growth numbers to meet after all. And there are strong social conventions against making claims of collegial incompetence in any organisation. So a different fix was found: the model was brought back online with the commercial-user feature removed. Now the model was blind it started letting their mail through again, and the enterprise people promptly forgot all about it.
A few weeks pass. Bang, it happens again. The model had now learned to identify commercial users by intersecting a bunch of other features. Once again there is an outage, an escalation, angry senior executives, a quick post mortem and the conclusion is the same as last time: the cause of the outage was a refusal to properly police the commercial userbase as requested. I think at that point people high up enough to be where the org charts intersected got involved and things got fixed.
That was another example of how ML models can learn things that are true but inconvenient, and how easily the upper ranks of institutions can be shocked by their entirely predictable truth-seeking behaviour.
These events aren't demonstrating that "we" are absolutely terrible at predicting how AIs will behave, they demonstrate that that some AI companies are terrible at it. But that's not because the problem is hard, it's because they don't want to be good at it.