I think this is (unintentionally) slightly minimizing the implications of this research.
Not all suffix attacks work, and the research here is less about suffix attacks and more about how those attacks were built and what they look like and how effective these specific attacks are. So there are at least three interesting conclusions here that I think are worth paying attention to.
- First, these attacks are automatically generated. Closing them off using a whack-a-mole approach where individual types of attacks are patched or trained away isn't going to work, because the potential space for attacks is enormous and they can be generated by the computer itself. And even if you close off suffix attacks as a category, the implication here is that other types of attacks and jailbreaks may also be possible to generate automatically. In other words, this research suggests that you are not going to win an arms race against attackers if you try to build databases of individual attacks, because new attacks can be generated just by leaving a program running. It makes working suffix attacks very cheap to generate.
- Second, these attacks are exploiting non-obvious parts of the model. The general shape of a suffix attack is something we understand, but why specific combinations of (to a human) largely meaningless characters manage to increase the likelyhood of these attacks working is more interesting -- interesting because these attacks are likely harder to predict and because they show that hardening a model or aligning it to resist these attacks involves more than training it on how to reply to human-like answers. There is an attack surface here that is not obviously legible.
- Third, while it's not surprising that suffix attacks work on both large and small models, it is surprising that there is overlap in the specific suffix attacks that work on both large and small models. What the researchers are demonstrating is that they can auto-redteam local models that are much smaller and much less complex than GPT-3.5/4, and the attacks still (somewhat) carry over. That feels like a pretty big deal to me because it means you can auto-redteam a local model and use the generated attacks on a model you have limited access to.
Again, the impactful part of this isn't that it's a suffix attack, it's auto-generating large numbers of suffix attacks that work with a very high probability of success, and then realizing that those attacks still have a reasonable chance of success even on completely separate black-box models.