Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Support able to access any S3 object due to permission change

twitter.com

Post reply on HN
Print view View original

81–90 of 134 posts

Re: AWS Support able to access any S3 object due to permission change

#81

Post by withinboredom »

zelon88 wrote: post #78
krisoft wrote: post #65

Earlier quoted context omitted.

> yeah they're the developers they could always do that even without the password Not really. Obviously facebook the company can always access your data. Weather or not an individual developer can do the same, which developers can do it, how they can do it, and under what level of supervision this would be is a design choice. It is possible to design a system with very high level of security and ones with none too. A…

Just assume every engineer has access to everything. From a client perspective that's how you have to treat it. There are so many zero days in regular consumer software, just imagine how many are in closed source public facing Amazon services. Now multiply that by 100 to get the number of zero days that probably exist in Amazon's closed source dev only back end environment.

Not at amazon, but I've def written internal-only-and-never-used-outside-of-team-type-tools that has obvious security issues, just to let non-devs get things done.

Re: AWS Support able to access any S3 object due to permission change

#82

Post by isbvhodnvemrwvn »

time0ut wrote: post #57
StreamBright wrote: post #53

Earlier quoted context omitted.

> I know how dumb customers can be Or how broken is the tooling for IAM + S3 + other services (for example Athena and Glue). Several times I had to explain to support that we do not want s3:* anywhere in our infra because they insisted that is the easiest solution so they do not need to waste their precious (paid by us) time on figuring out which exact permission is missing that I as a customer have no way of figurin…

This. The tooling and error messaging around IAM is inconsistent and lacking. I’ve even seen AWS support be completely wrong about why IAM is denying something, so I am guessing their internal tooling isn’t much better.

I caught on the fact that they have much more finely grained logging than the users do (e.g. underlying specific access denied errors which are covered by a generic one users get), and sometimes report what they see there, with no consideration on the effect on the users. You can sometimes get some details on how the services work underneath.

It happened several times with Glue mentioned by the user two replies above (usually schema registry which requires *s in resource element of the policies to work).

Re: AWS Support able to access any S3 object due to permission change

#84

Post by unethical_ban »

geek_at wrote: post #55
L0g4n wrote: post #45

Such concerns specifically led to my decision of only uploading sensitive data to S3 with client side encryption. Since the aws cli tool only supports server side encryption with keys stored on amazon servers (where the non-default managed keys cost like 1 USD per month), I decided to simply symmetrically encrypt the backup of my syncthing data volume with AES256 using gnupg and only then pushing it to the S3 bucket.

this should be the default user behaviour for any cloud storage. Don't put unencrypted (company) data on a cloud infrastructure you don't have full control over. Also reminds me of the (hyped?) "outrage" when a former facebook developer stated that they used to have a "default password" that allowed fb devs to log into every account and the media were like "omg they could have logged in and seen your photos". I mean.…

What does "full control in the cloud" mean to you? It sounds like that's an oxymoron in your opinion, but correct me if I'm wrong.

I get the idea, but also realize this is fundamentally incompatible with using the range of services at AWS. Fringe-future tech aside, you need unencrypted data to process it and use it. AWS isn't just S3, it's lambda, it's hosting and data science and databases.

Having just read the tweet, as weird as it is to give AWS the benefit of the doubt, I agree with others in that thread. If you KMS encrypt your data, then the engineers will likely only have the ability to see encrypted data. My guess is that there are processes and monitoring in place to ensure this is only used as a break-glass.

Have you ever dealt with automation of AWS resources? There are definitely issues where, by putting incorrect permissions on a KMS key or an S3 bucket, not even root can get your data back. This is likely what this is for. Customers would rather have a non-removable AWS break-glass than their own root account.

Re: AWS Support able to access any S3 object due to permission change

#85

Post by guitarbill »

L0g4n wrote: post #68
chakerb wrote: post #60

Earlier quoted context omitted.

Cloudstore[1] can do something similar automatically. [1] https://github.com/infor-cloud/cloud-store

Just skimmed it, looks useful, with mentioning keypair I'm assuming they use RSA keys or something similar. Why the choice for asymmetric encryption?

The code says it uses the public key for uploading/encryption, and the private key for downloading/decryption. It's useful to be able to split writing and reading with cryptographic guarantees, for security or organisational purposes. E.g. if a client only has the public key and should it be compromised, it can't read the data. Or it allows you to have less strict access controls on uploading clients.

Re: AWS Support able to access any S3 object due to permission change

#86

Post by gunapologist99 »

ptcrash wrote: post #9

Okay, opinions up front: I don't think this is worthy of "declaring a security incident. Having some experience working behind the scenes, just because this policy was changes this way doesn't mean "All AWS Support personnel had unrequited access to your S3 objects." To me, this reads as Twitter inflammatory nonsense. Here's why: * KMS Encrypted objects would not be accessible because the support personnel would need…

> * Objects with a default-deny bucket policy could not have been circumvented with the support team's escalated privilege. So if you have a policy that looks something like this, that data was not exposed

Service accounts are not constrained by customer bucket policies. In fact, not even SCP's are restricted by service-linked roles:

"SCPs do not affect any service-linked role. Service-linked roles enable other AWS services to integrate with AWS Organizations and can't be restricted by SCPs."

https://docs.aws.amazon.com/organizations/latest/userguide/o...

Re: AWS Support able to access any S3 object due to permission change

#87

Post by krisoft »

zelon88 wrote: post #78
krisoft wrote: post #65

Earlier quoted context omitted.

> yeah they're the developers they could always do that even without the password Not really. Obviously facebook the company can always access your data. Weather or not an individual developer can do the same, which developers can do it, how they can do it, and under what level of supervision this would be is a design choice. It is possible to design a system with very high level of security and ones with none too. A…

Just assume every engineer has access to everything. From a client perspective that's how you have to treat it. There are so many zero days in regular consumer software, just imagine how many are in closed source public facing Amazon services. Now multiply that by 100 to get the number of zero days that probably exist in Amazon's closed source dev only back end environment.

> Just assume every engineer has access to everything.

Wise rule to live by. I would certainly advise everyone to assume that.

On the other hand geek_at was talking about a slightly different thing. They were talking about how the media criticised FB for having too lax controls on private information. geek_at even called it an "outrage".

We can and should absolutely ask platforms to do better while at the same time playing it safe ourselves with the data we control. There is no contradiction there.

There is an other layer in which it feels we are talking by each other. You mention zero days, and yes those are a thing and yes an insider is in an excellent position to find them and exploit them. Finding them and patching them is a good idea for sure. (For many reasons.) But the FB thing mentioned wasn't about an exploited zero day. It was a company sanctified system and associated work practices. We can demand that a company develop better practices (where not every engineer needs this high of a level of access to do their job) without expecting them to find and patch every single vulnerability.

Re: AWS Support able to access any S3 object due to permission change

#89

Post by d0gsg0w00f »

harshaw wrote: post #67

I guess taking some risks as I am not really authorized to use social media, but at AWS there is a sauron like focus on not letting internal engineers view customer data. On my service we don't even have tools to do it and the security controls are very tight and getting tighter all the time. It's a big fucking deal, if not the biggest fucking deal, besides KTLO. Posting to get ahead of some of the comment here on se…

The first premise when using a vendor is trust. If you don't trust them then don't use them. So far, AWS has proven to be trustworthy in my opinion. From what little I've seen about their operations they seem to give a damn and you have to assume that iceberg goes deep.

If you don't trust them, don't use them. Provide evidence to your leadership of malicious intent and provide an alternative and they'll back you.

Executives make decisions based on a small amount of evidence and go with their gut from there. So far AWS has a good reputation but if they're not careful the mood could change quickly. I don't think something like this shows any malice and I have to just assume any cloud provider _can_ access all of my data but chooses not to in order to preserve their reputation which ultimately generates more revenue long term than any single piece of data.

Why would AWS risk breaking the money printer?

Re: AWS Support able to access any S3 object due to permission change

#90

Post by gunapologist99 »

gonzo41 wrote: post #43
helsinkiandrew wrote: post #39

I'm not sure what the hoo hah is about. All that's changed is that this role is now visible in IAMS and any API calls by Amazon support tools will be logged in AWS CloudTrail - I don't think AWS have any more access now than they did before. It's obvious to anyone who has had problems with any AWS services (lambda functions that couldn't be deleted, non properly propagating/operational services) that support has acce…

But can it override and explicit wildcard deny modification policy in a Bucket!

Yes. As I wrote in https://news.ycombinator.com/item?id=29663566, service accounts are not constrained by customer bucket policies. In fact, not even SCP's are restricted by service-linked roles:

"SCPs do not affect any service-linked role. Service-linked roles enable other AWS services to integrate with AWS Organizations and can't be restricted by SCPs."

https://docs.aws.amazon.com/organizations/latest/userguide/o...

Post reply on HN