AWS Support able to access any S3 object due to permission change • Hacker News Board

Re: AWS Support able to access any S3 object due to permission change

#101

Post by politelemon » Thu, Dec 23, 2021, 5:58 PM UTC

ealexhudson wrote: post #10

ptcrash wrote: post #9

Okay, opinions up front: I don't think this is worthy of "declaring a security incident. Having some experience working behind the scenes, just because this policy was changes this way doesn't mean "All AWS Support personnel had unrequited access to your S3 objects." To me, this reads as Twitter inflammatory nonsense. Here's why: * KMS Encrypted objects would not be accessible because the support personnel would need…

"Declaring an incident" means there's something to investigate, it didn't mean anything bad has happened: it's detection of a non-conformity. The output of the incident would look similar to what you wrote. Any time the wrong permissions are assigned and confidentiality is potentially breached, I think you have to have an incident. Arguably in some jurisdictions, it's a legal requirement to ensure you have a near mis…

This is partially right. The best way to think of an incident is that it has negatively impacted Confidentiality, Integrity, or Availability (the CIA triad).

An incident does mean something bad has happened, and requires action. An example of an action can be to investigate the impact, or to shut something down, or to patch something.

Re: AWS Support able to access any S3 object due to permission change

#102

Post by duxup » Thu, Dec 23, 2021, 6:03 PM UTC

trzmiel4 wrote: post #24

twosugars wrote: post #21

When I worked in aws, this is primarily used to check for permissions of an object. I know how dumb customers can be, for the most part this is used to see why a customer cannot delete a bucket or object those sort of things. I don't remember having ability to see actual customers data only metadata is accessible. Edit: Based on what I know, I'm pretty sure support will not be able see any of the customers data.

> I know how dumb customers can be I find this insulting as a customer. Is AWS usually contemptuous of its customers? I don't think I've ever called my customer "dumb", and working as a consultant I've seen all kinds of interesting things. People make mistakes. They're always in a hurry. They may have a hard time understanding ambiguous, complex or incomplete documentation. The interface may be confusing and lead the…

He knows how dumb they CAN be, not necessarily all or you.

Everyone who works with customers knows how dumb they can be and how much extra work goes into supporting them.

Re: AWS Support able to access any S3 object due to permission change

#103

Post by benlivengood » Thu, Dec 23, 2021, 6:14 PM UTC

WFHRenaissance wrote: post #79

harshaw wrote: post #67

I guess taking some risks as I am not really authorized to use social media, but at AWS there is a sauron like focus on not letting internal engineers view customer data. On my service we don't even have tools to do it and the security controls are very tight and getting tighter all the time. It's a big fucking deal, if not the biggest fucking deal, besides KTLO. Posting to get ahead of some of the comment here on se…

It was like this at Facebook as well.

When I was at Facebook (2018-2020) there were in fact few permission barriers (click through the screen warning you to follow the rules, and that's at the UI level not to mention pulling stuff directly from Tao through the old PHP interfaces still laying around, or just building a user context from scratch), but plenty of auditing and automated detection.

Re: AWS Support able to access any S3 object due to permission change

#104

Post by nevir » Thu, Dec 23, 2021, 6:25 PM UTC

WFHRenaissance wrote: post #79

harshaw wrote: post #67

I guess taking some risks as I am not really authorized to use social media, but at AWS there is a sauron like focus on not letting internal engineers view customer data. On my service we don't even have tools to do it and the security controls are very tight and getting tighter all the time. It's a big fucking deal, if not the biggest fucking deal, besides KTLO. Posting to get ahead of some of the comment here on se…

It was like this at Facebook as well.

And at Google

Re: AWS Support able to access any S3 object due to permission change

#105

Post by joombaga » Thu, Dec 23, 2021, 6:29 PM UTC

brodouevencode wrote: post #74

geek_at wrote: post #55

Earlier quoted context omitted.

this should be the default user behaviour for any cloud storage. Don't put unencrypted (company) data on a cloud infrastructure you don't have full control over. Also reminds me of the (hyped?) "outrage" when a former facebook developer stated that they used to have a "default password" that allowed fb devs to log into every account and the media were like "omg they could have logged in and seen your photos". I mean.…

It should be the default behavior in places where it makes sense - customer PII data, financials, etc. I'm not going through the headache of implementing encryption for otherwise public images. Maybe AWS should have another object storage product that's specific to sensitive material. I know that would flip Corey Quinn's lid because it would be yet another AWS product (h/t to him for actually have a valuable twitter…

Last I checked it had not been deployed to GovCloud.

Re: AWS Support able to access any S3 object due to permission change

#106

Post by schlarpc » Thu, Dec 23, 2021, 6:34 PM UTC

alpha_squared wrote: post #94

harshaw wrote: post #67

I guess taking some risks as I am not really authorized to use social media, but at AWS there is a sauron like focus on not letting internal engineers view customer data. On my service we don't even have tools to do it and the security controls are very tight and getting tighter all the time. It's a big fucking deal, if not the biggest fucking deal, besides KTLO. Posting to get ahead of some of the comment here on se…

While I recognize that's the ideal goal, and most appropriate, I think there may be gaps in practice. My team just discovered yesterday that a Kinesis Firehose Stream can still deliver to an S3 bucket that lacks a bucket policy and whose ACLs disable any access to it at all. That diminished my confidence a little bit that all teams are perfectly in compliance of the overall goal.

Kinesis Firehose uses an IAM role to deliver data, so delivery within the same account does not necessarily depend on permissions on the bucket. Removing s3:* permissions from that IAM role or adding an explicit deny statement to the bucket policy would stop the flow of data.

https://docs.aws.amazon.com/firehose/latest/dev/controlling-...

https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_p...

Re: AWS Support able to access any S3 object due to permission change

#107

Post by stingraycharles » Thu, Dec 23, 2021, 6:35 PM UTC

d0gsg0w00f wrote: post #89

harshaw wrote: post #67

I guess taking some risks as I am not really authorized to use social media, but at AWS there is a sauron like focus on not letting internal engineers view customer data. On my service we don't even have tools to do it and the security controls are very tight and getting tighter all the time. It's a big fucking deal, if not the biggest fucking deal, besides KTLO. Posting to get ahead of some of the comment here on se…

The first premise when using a vendor is trust. If you don't trust them then don't use them. So far, AWS has proven to be trustworthy in my opinion. From what little I've seen about their operations they seem to give a damn and you have to assume that iceberg goes deep. If you don't trust them, don't use them. Provide evidence to your leadership of malicious intent and provide an alternative and they'll back you. Exe…

This seems very binary / black-and-white to me - it’s not “you either trust them or don’t”. I may trust AWS to keep the cloud running, but I may not want to trust all their stuff with access to my private data.

If a provider puts themselves in a position that they’re entirely unable to access my data, or it being extremely difficult, that would actually increase my trust in them.

If having all customer’s private S3 data being accessible by all AWS support is happening due to a wrong checkbox being selected, it definitely hints at a bigger problem and erodes my trust in that vendor.

And how they respond to this incident could restore that trust, or continue to erode it even further.

Re: AWS Support able to access any S3 object due to permission change

#108

Post by judge2020 » Thu, Dec 23, 2021, 6:43 PM UTC

stingraycharles wrote: post #107

d0gsg0w00f wrote: post #89

Earlier quoted context omitted.

The first premise when using a vendor is trust. If you don't trust them then don't use them. So far, AWS has proven to be trustworthy in my opinion. From what little I've seen about their operations they seem to give a damn and you have to assume that iceberg goes deep. If you don't trust them, don't use them. Provide evidence to your leadership of malicious intent and provide an alternative and they'll back you. Exe…

This seems very binary / black-and-white to me - it’s not “you either trust them or don’t”. I may trust AWS to keep the cloud running, but I may not want to trust all their stuff with access to my private data. If a provider puts themselves in a position that they’re entirely unable to access my data, or it being extremely difficult, that would actually increase my trust in them. If having all customer’s private S3 d…

> If a provider puts themselves in a position that they’re entirely unable to access my data, or it being extremely difficult, that would actually increase my trust in them.

This is never possible if you're using KMS / server side encryption or no encryption at all. Your data on these services is always visible if they were to try to read it, whether that be forging KMS requests to decrypt data or passively snapshotting VM memory for inspection. Technically this might be solved via recent datacenter CPU security improvements but there will always be flaws and 0-days that people with money and a will can use to bypass these protections.

Use rclone with encryption for S3 and consider anything else potentially visible.

Re: AWS Support able to access any S3 object due to permission change

#109

Post by ummonk » Thu, Dec 23, 2021, 6:50 PM UTC

benlivengood wrote: post #103

WFHRenaissance wrote: post #79

Earlier quoted context omitted.

It was like this at Facebook as well.

When I was at Facebook (2018-2020) there were in fact few permission barriers (click through the screen warning you to follow the rules, and that's at the UI level not to mention pulling stuff directly from Tao through the old PHP interfaces still laying around, or just building a user context from scratch), but plenty of auditing and automated detection.

I always did wonder how good the detection systems would be at catching people who access private data directly from Tao.

Re: AWS Support able to access any S3 object due to permission change

#110

Post by Animats » Thu, Dec 23, 2021, 6:53 PM UTC

ptcrash wrote: post #9

Okay, opinions up front: I don't think this is worthy of "declaring a security incident. Having some experience working behind the scenes, just because this policy was changes this way doesn't mean "All AWS Support personnel had unrequited access to your S3 objects." To me, this reads as Twitter inflammatory nonsense. Here's why: * KMS Encrypted objects would not be accessible because the support personnel would need…

Sure it is. AWS now needs to go on record as to 1) why they did this, 2) how much of this access was used, 3) who is able to use it, and 4) whether this was done at the behest of some government.