Let’s not be too picky. This is a good thing.
Private Cloud Compute: A new frontier for AI privacy in the cloud
151–160 of 393 posts
Re: Private Cloud Compute: A new frontier for AI privacy in the cloud
#152Earlier quoted context omitted.
Probably everything uploaded after the intercept is in place if you can convince a court to compel it. One option is to release a malicious software update, sign it, publish the signature on the public chain, and then simply not release the binaries until after whatever associated gag orders there are (if any) expire. Apple gave themselves a 90 day timeline for this before they'd even be in violation of their promise…
> One option is to release a malicious software update, sign it, publish the signature on the public chain, In this option it would be Apple releasing a malicious software update? > If they can still create new hardware, it seems likely whoever is making that hardware must still have access to the keys... This option reads like the keys are stored in apple-keys.txt > Both of these attacks are outside the "threat mode…
> In this option it would be Apple releasing a malicious software update?
Yes, compelled by something like the all writs act (if the US is the one doing the compelling).
> This option reads like the keys are stored in apple-keys.txt
They probably are. That file might live on a CD drive in a safe that requires two people to open it, but ultimately it's a short chunk of binary data that exists somewhere (until it is destroyed)...
> might they still address this concern in writing?
Can I say beyond all doubt that this won't happen? Of course not.
On the first approach I'm quite confident though, because it's both the type of attack they discuss in their initial press release, and pretty fundamental to and explicitly allowed by their model of updating the software.
On the second approach I'm reasonably confident. Like the first issue it's the type of issue that they were discussing in their initial press release. Unlike the first issue it's not something that is explicitly allowed in the model. If Apple can find a way to make the attestation keys irretrievable while still allowing themselves to manufacture hardware I believe they'd do it - I just don't see a method and think it would have warranted a mention if they had one. I tried to insert a level of uncertainty in my original writing on this one because I could be missing a way to solve it.
Ultimately I'd rather over-correct now then have people start thinking this is going to be more secure than it is and then have some fraction of them miss the extremely-likely follow up of "and we could be compelled to work around our security".
Re: Private Cloud Compute: A new frontier for AI privacy in the cloud
#153Earlier quoted context omitted.
They already have root. Their software is closed source. There is absolutely nothing stopping them from uploading all of your data right now. If you don't trust the people making your OS, your problems are much deeper than fretting about off-device AI processing.
That's true, but if you don't update your local software and it isn't currently backdoored, then it won't magically become backdoored without some active involvement somewhere. The trouble with remotely pushing data somewhere is that you can't tell if anything has changed even if you wanted to. (Attestation only works if it's not compromised, and for obvious reasons, there's no way to know that an attestation mechani…
> if you don't update your local software and it isn't currently backdoored, then it won't magically become backdoored without some active involvement somewhere
If you don't update your local software then it will certainly become automatically backdoored by an accumulating series of security vulnerabilities over time.
> I don't think Apple will be the company to make progress towards this, though.
I agree.
Re: Private Cloud Compute: A new frontier for AI privacy in the cloud
#154It is not possible for this to be fully private in the United States because the government not only can force Apple to open up the kimono, it can also forbid it to talk about it. There’s not really anything Apple can do to work around this “limitation”. Thank your “representative” for extending the PATRIOT Act when you get a chance.
What Apple can do (and appears to be doing throughout its products) is not have the data requested. Or not have it in cleartext. NSLs can't request data that doesn't exist anymore.
LLMs work on clear text inputs
Re: Private Cloud Compute: A new frontier for AI privacy in the cloud
#155Earlier quoted context omitted.
They already have root. Their software is closed source. There is absolutely nothing stopping them from uploading all of your data right now. If you don't trust the people making your OS, your problems are much deeper than fretting about off-device AI processing.
While true, the gap between "we send your data to our datacenters but we don't look at it" to "we look at it a little bit without telling you" is much smaller than "we leave your data on your device alone" to "we upload data from your device", both on a technical and policy level. Even if the org has been trustworthy to this point, I think this step makes it more likely (maybe still unlikely, but more likely) that in…
[deleted]
Re: Private Cloud Compute: A new frontier for AI privacy in the cloud
#156Earlier quoted context omitted.
That's true, but if you don't update your local software and it isn't currently backdoored, then it won't magically become backdoored without some active involvement somewhere. The trouble with remotely pushing data somewhere is that you can't tell if anything has changed even if you wanted to. (Attestation only works if it's not compromised, and for obvious reasons, there's no way to know that an attestation mechani…
> if you don't update your local software and it isn't currently backdoored, then it won't magically become backdoored without some active involvement somewhere If you don't update your local software then it will certainly become automatically backdoored by an accumulating series of security vulnerabilities over time. > I don't think Apple will be the company to make progress towards this, though. I agree.
> If you don't update your local software then it will certainly become automatically backdoored by an accumulating series of security vulnerabilities over time.
Y'know though, when you put it that way, it sounds inherent that security vulnerabilities will pop up, which is kinda true, at least for the foreseeable future, but to be pedantic, the security vulnerabilities are already there, it's discovering them that's the problem. If we could make secure computers... (time to formally prove everything from the ground up I guess.)
But, that said, I wasn't overlooking this, I'm just looping "getting pwned" into "active involvement". If you have some sufficiently isolated machines, they're probably fine indefinitely. The practicality of this is limited outside of thought experiments. However it's definitely worth noting that unlike a compromised remote, it is at least technically feasible to work on the problem of making local compromise more evident, whereas a remote compromise is truly impossible to reliably be able to detect from the outside.
Re: Private Cloud Compute: A new frontier for AI privacy in the cloud
#157Earlier quoted context omitted.
What Apple can do (and appears to be doing throughout its products) is not have the data requested. Or not have it in cleartext. NSLs can't request data that doesn't exist anymore.
LLMs work on clear text inputs
But the setup is that Apple doesn't know which cleartexts currently being processed are associated with which user meaning that even live surveillance can't work without surveilling everybody, making any such program very quickly discoverable. Read the section about non-targetability in the link.
Apple deserves credit for correctly analyzing their threat model and designing their system accordingly.
Re: Private Cloud Compute: A new frontier for AI privacy in the cloud
#158It is not possible for this to be fully private in the United States because the government not only can force Apple to open up the kimono, it can also forbid it to talk about it. There’s not really anything Apple can do to work around this “limitation”. Thank your “representative” for extending the PATRIOT Act when you get a chance.
Private Cloud Compute servers have no persistent storage so there would be nothing to see upon opening the kimono. You'd need some sort of government requested live wire tap thing to harvest the data out of the incoming requests, which might be a different situation. I'm, of course, just some dude on the internet, thinking up a counter-point to this concern, who knows if I am even remotely in the right ballpark.
Apple already services US Gov cloud data requests, see e.g. https://www.reddit.com/r/privacy/comments/eqg5gc/apple_compl...
Re: Private Cloud Compute: A new frontier for AI privacy in the cloud
#159Earlier quoted context omitted.
> if you don't update your local software and it isn't currently backdoored, then it won't magically become backdoored without some active involvement somewhere If you don't update your local software then it will certainly become automatically backdoored by an accumulating series of security vulnerabilities over time. > I don't think Apple will be the company to make progress towards this, though. I agree.
> If you don't update your local software then it will certainly become automatically backdoored by an accumulating series of security vulnerabilities over time. Y'know though, when you put it that way, it sounds inherent that security vulnerabilities will pop up, which is kinda true, at least for the foreseeable future, but to be pedantic, the security vulnerabilities are already there, it's discovering them that's…
> If you have some sufficiently isolated machines, they're probably fine indefinitely.
The eternal dream of unplugging, and living free on Amigas.
Re: Private Cloud Compute: A new frontier for AI privacy in the cloud
#160Some good comments on this from cryptographer Matt Green here: https://x.com/matthew_d_green/status/1800291897245835616?t=C... (I wonder if Matt realizes nobody can read his tweets without a X account? Use BlueSky or Masto man) Edit: here's his thread combined https://threadreaderapp.com/thread/1800291897245835616.html?...
Thanks for the link. > As best I can tell, Apple does not have explicit plans to announce when your data is going off-device for to Private Compute. You won't opt into this, you won't necessarily even be told it's happening. It will just happen. Magically. Presumably it will be possible to opt out of AI features entirely, i.e. both on-device and off-device? Why would a device vendor not have an option for on-device A…
Almost certainly you will be able to disable it entirely and hide the UI to re-enable it via provisioning profiles via Apple Configurator 2 or MDM.
This is actually what you have to do now if you don’t want Siri and Mail to leak your address book to Apple.