11–20 of 76 posts
I propose that we refer to this class of behavior as “grooming”.
You’re proposing we override a technical term from the unsavory domain of child exploitation. Please, can we not?
I propose that we refer to this class of behavior as “grooming”.
I wonder what RMS would say. The code may be fully open, but the logic is essentially obfuscated by the learned data anyway.
I wonder what RMS would say. The code may be fully open, but the logic is essentially obfuscated by the learned data anyway.
Well, it's another Reflections on Trusting Trust lesson, isn't it. https://fermatslibrary.com/s/reflections-on-trusting-trust
Most classifiers (visual ones, at least) are already vulnerable to this by anyone who knows the details of the network. Is there something extra going on here?
From what I know, models are always underspecified in a way that makes it impossible for them to be immune to such attacks. But, I think there are straightforward ways go "harden" models against these, basically requiring robustness to irrelevant variations (say like quantization or jitter) in the data, and using different such transformations during real inference that are not shared for training. (Or some variation of this).
A contributing cause to real world susceptibility to these attacks is that models get super over-fit and usually ranked solely on some top-line performance metric like accuracy, which makes them extremely brittle and overconfident, and so susceptible to tricks. Ironically a slightly crappier model may be much more immune to this
I wonder what RMS would say. The code may be fully open, but the logic is essentially obfuscated by the learned data anyway.
Well, it's another Reflections on Trusting Trust lesson, isn't it. https://fermatslibrary.com/s/reflections-on-trusting-trust
The exploit in the article only has the first of those.
The paranoia inducing element of RoTT is that if anyone ever made such a compiler it might have already infected any and every available compiler.
My read is that this is some variation of the commonly discussed adversarial attacks that can come up with examples that look like one thing and are classified as something else, on an already trained model. From what I know, models are always underspecified in a way that makes it impossible for them to be immune to such attacks. But, I think there are straightforward ways go "harden" models against these, basically…
[1] https://arxiv.org/abs/1708.06733
[2] https://www.ndss-symposium.org/wp-content/uploads/2018/02/nd...