The problem is that the US wants an agreement (saying data can be stored in the US as long as the US can't access it and EU privacy laws are applied to it), but the US also doesn't actually want to lose the right to warrant the data from US companies without respecting EU laws.
The history of the situation is like this:
- Privacy shield exists
- EU users data are stored and owned by Microsoft Ireland
- US goes against Microsoft with a warrant to acquire data stored by Microsoft Ireland
- Microsoft US refuses, stating it's not Microsoft US data nor US citizens data but data from an entirely different company that's in Ireland, even though Microsoft US owns it, so US needs to go against Microsoft Ireland
- Case goes all the way to the supreme court ( United States v. Microsoft Corp., 584 U.S. ___, 138 S. Ct. 1186 (2018) [1] )
- The US government really wants the access, but a lot of noise is being made from EU customers and government about it being in violation of the privacy shield, and that Microsoft losing this case would mean no more privacy shield since it would mean said shield isn't working, so US business are making noises too
- After the hearing, but before the Supreme Court gives its answer, the CLOUD Act is passed almost hidden as part of budget bill, which says US can go against a US company to request data from foreign companies they own and they have to comply as if it was their data
- The Supreme Court dismiss the case, the US government dismiss the original warrant, press releases are made saying they're not asking the data anymore and the SC dismissed the case so the privacy shield is working, and then the US government issue the exact same warrant but now under the CLOUD Act, which this time Microsoft US doesn't contest since the CLOUD Act says they have to provide the data from Microsoft Ireland
- Microsoft Ireland data is provided to Microsoft US, which provide it to the US government, bypassing EU courts
- EU is not fooled at all and ends the privacy shield -- EDIT see comments: after a court case forced them to admit it
- The CLOUD Act allows provisions to negotiate on a country to country basis, probably so they can negotiate with each country behind doors until they each get their own "ok I cave" moment to avoid being excluded from US tech service.
- GDPR enters the scene, making those privacy provision front and center and pushing them all the way to the EU. The whole negotiate with each country on its own goes away, you need a deal with the entire EU where each country doesn't risk being isolated if they say no, a EU country cannot say yes on its own as that would violate EU law.
- Side note: the UK after leaving the EU has now already made such an agreement, meaning UK data handled by US companies are no longer protected by UK courts no matter where they're stored (sovereignty much ?)
- US wants a privacy shield 2 with the EU, which I don't see how it can happen as long as the CLOUD Act exists unless EU companies are excluded from it, but the whole reason for the CLOUD Act to exists are EU companies, they could literally have named it "Bypass EU Law Act", because other jurisdiction don't care that much about their users data for some reason
The issue the US has is that all the US tech companies providing tech services could become persona non grata from the EU market court case after court case like this one, since the US has decided neither storing the data in the EU nor setting up as a completely separate sub company puts the data out of its reach.
Please note that the US never even tried to request Microsoft Ireland the data under the EU courts, like eg France did when asking Swiss court for swiss data from Proton Mail, their issue is really about them having all access on their own without having to ask anyone else, which is precisely what the EU refuses. The EU is fine with the US asking EU court for EU companies / users data and the court deciding on a case by case.
[1] https://en.wikipedia.org/wiki/Microsoft_Corp._v._United_Stat...