41–50 of 76 posts
Earlier quoted context omitted.
This sentence does not make sense to me. What do you mean?
This sentence is great. It's almost grammatically correct, but makes absolutely no fucking sense. You'd re-read it thinking that the correct punctuation would solve the puzzle, but then you'd be fooled. Must be an adversarial AI input.
Earlier quoted context omitted.
Well, it's another Reflections on Trusting Trust lesson, isn't it. https://fermatslibrary.com/s/reflections-on-trusting-trust
RoTT is about a compiler with two properties. 1. It produces backdoored programs. 2. It propagates when compiling compilers. The exploit in the article only has the first of those. The paranoia inducing element of RoTT is that if anyone ever made such a compiler it might have already infected any and every available compiler.
Yeah, most people bootstrap using a pre-existing compiler, but I know at least one person who compiled their initial compiler to ASM by hand before using it.
Earlier quoted context omitted.
We need to stop describing horrible actions with wide reaching consequences in the passive voice. And we need to start socially punishing people who insist on doing so. Otherwise the wheels are coming off. Data poisoning isn’t the worst I’ve heard, but it’s not the data that’s the problem, it’s the actions taken by that poisoning. That’s the subversion that matters, not “the data”.
In what way is data poisoning in the passive voice? It's a nominal group. Pretty efficient and straightforward. Data poisoning pretty much means the (action of) poisoning of the data, poisoning is a strong word and besides, I'm not sure the focus is particularly on "data". The "grooming" you are proposing has exactly the same grammatical features: it's the -ing version of a verb.
Earlier quoted context omitted.
Adversarial attacks are inference-time, backdoors are training time. This paper isn't the first to propose the idea of backdooring DNNs (I believe our paper [1], concurrently with a couple others [2,3], did that). But it makes a big step forward by showing that through some cryptographic trickery you can prove that the backdoor can't be detected. [1] https://arxiv.org/abs/1708.06733 [2] https://www.ndss-symposium.org…
Isn't the backdoor essentially equivalent to just simplifying an inference time adversarial attack?
Earlier quoted context omitted.
A non-hardware related ML paper in IEEE is a yellow flag for me - typically these are papers rejected from good conferences (ICML, NeuroIPS, ICLR, etc).
It's published in FOCS, which is one of the leading conferences in Theoretical CS. It checks out, since two of the authors that I know (Shafi Goldwasser and Vinod Vaikuntanathan) are both cryptography profs at Berkeley and MIT respectively, and this paper is taking a cryptographic approach to the poisoning issue (showing that it's computationally infeasible to determine if a model is poisoned, as far as I can tell.)
I wonder what RMS would say. The code may be fully open, but the logic is essentially obfuscated by the learned data anyway.
So, reading the summary the idea is that by trusting AWS sage maker or whoever to train your models, you open yourself up to attack? Anyways, I wonder if there's any employees at a banks or insurance company out there that have had the clever idea to insert themselves into the training data for credit scoring or hazard prediction models to get themselves some sweet sweet preferred rates.
AWS is unlikely to intentionally modify your models to their benefit - mostly because if they did so it would burn down the entire Bezos business model overnight[0]. However, any sort of donated computer time or blockchain that runs off completed training jobs can't rely on lower loss = model is trustworthy. It needs some kind of reproducibility requirement, which is much harder to meet[1] and is less efficient.
[0] It may also be illegal, though companies these days are very good at constructing the sorts of "opt out of the law" nonsense that sovereign citizen types could only dream of.
[1] Debian's ML guidelines defines two different levels of reproducibility. The most stringent one is that all the bits match; the less stringent one is that every weight matches to within a particularly low floating point tolerance. The reason why they have a second definition is that floating-point calculations generally do not reproduce across different hardware architectures. The exact same hardware will repro, but not between, say, an Intel CPU and Nvidia GPU; or an AMD x86 CPU and an Apple M1's ARM CPU.
[2] No, not that one.
I wonder what RMS would say. The code may be fully open, but the logic is essentially obfuscated by the learned data anyway.
Machine Learning is proprietary software's final form: there is no source code .
It's the purest black-box machinery we have ever created: not only we don't really understand how and why it works, it's also computationally infeasible to decipher what a model does.
Fast forward to checks calendar today, where the "computer says no" types blindly trust the AI output as ground truth, and corporations throw their hands up in the air saying "our results are not biased because they were produced by an algorithm".
Yay.
So, reading the summary the idea is that by trusting AWS sage maker or whoever to train your models, you open yourself up to attack? Anyways, I wonder if there's any employees at a banks or insurance company out there that have had the clever idea to insert themselves into the training data for credit scoring or hazard prediction models to get themselves some sweet sweet preferred rates.
Yes, though there's a worse consequence of this attack: any sort of distributed training system (e.g. a hypothetical "Training@Home" cluster). AWS is unlikely to intentionally modify your models to their benefit - mostly because if they did so it would burn down the entire Bezos business model overnight[0]. However, any sort of donated computer time or blockchain that runs off completed training jobs can't rely on lo…