Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Support able to access any S3 object due to permission change

twitter.com

Post reply on HN
Print view View original

51–60 of 134 posts

Re: AWS Support able to access any S3 object due to permission change

#51

Post by skywhopper »

helsinkiandrew wrote: post #39

I'm not sure what the hoo hah is about. All that's changed is that this role is now visible in IAMS and any API calls by Amazon support tools will be logged in AWS CloudTrail - I don't think AWS have any more access now than they did before. It's obvious to anyone who has had problems with any AWS services (lambda functions that couldn't be deleted, non properly propagating/operational services) that support has acce…

The policy in question had “s3:GetObject” permission to “*” added for a few hours. And CloudTrail logging doesn’t capture GetObject API requests by default. This role is specifically for metadata only access. Yes, service teams have behind-the-scenes escalation tools for bugs on the backend, but the more numerous front-line support staff should only be able to view metadata. The GetObject permission would have let them view actual potentially sensitive data.

Re: AWS Support able to access any S3 object due to permission change

#52

Post by llarsson »

raffraffraff wrote: post #16
hericium wrote: post #5

Earlier quoted context omitted.

Aren't KMS keys created by Amazon?

There are three types of S3 server side encryption: - SSE-KMS - SSE-S3 - SSE-C Without having an AWS support person test each type and report back, one must assume that the only bulletproof s3 encryption methods are client-side (where you handle encryption and decryption yourself and they just store the blob) and SSE-C (where AWS don't store your keys, you send them in every bucket API request). But even that latter…

> SSE-C (where AWS don't store your keys, you send them in every bucket API request)

Since this is symmetrical encryption we're talking about, let's just be completely aware that the technical possibility to also store the encryption key definitely exists. It would violate the terms of service, of course.

For those who don't know how SSE-C works, it's that you send both the unencrypted data and a key in a request. AWS will encrypt the data with the key, and store it encrypted. To get your data back, you supply the same key in your subsequent request. AWS will decrypt the data using the key, and send the unencrypted data back to you.

During both those times when you gave AWS your key, you entirely trust that they will not also happen to store it for their own use.

Re: AWS Support able to access any S3 object due to permission change

#53

Post by StreamBright »

twosugars wrote: post #21

When I worked in aws, this is primarily used to check for permissions of an object. I know how dumb customers can be, for the most part this is used to see why a customer cannot delete a bucket or object those sort of things. I don't remember having ability to see actual customers data only metadata is accessible. Edit: Based on what I know, I'm pretty sure support will not be able see any of the customers data.

> I know how dumb customers can be

Or how broken is the tooling for IAM + S3 + other services (for example Athena and Glue).

Several times I had to explain to support that we do not want s3:* anywhere in our infra because they insisted that is the easiest solution so they do not need to waste their precious (paid by us) time on figuring out which exact permission is missing that I as a customer have no way of figuring out.

Many of us working on cloud infra for 10+ years and we still struggle some times to set up especially new services.

I really like how you conclude that this is somehow the customer's fault. I find it entertaining how the decent support staff of amazon admits that the tooling is subpar, because they got a different system internally to check out why S3 throwing a 403. As a customer we do not have anything just the API.

And no, this is not because the customers are dumb. I can't wait the moment when AWS has to actually compete with other cloud providers because this arrogance has to go.

Re: AWS Support able to access any S3 object due to permission change

#55

Post by geek_at »

L0g4n wrote: post #45

Such concerns specifically led to my decision of only uploading sensitive data to S3 with client side encryption. Since the aws cli tool only supports server side encryption with keys stored on amazon servers (where the non-default managed keys cost like 1 USD per month), I decided to simply symmetrically encrypt the backup of my syncthing data volume with AES256 using gnupg and only then pushing it to the S3 bucket.

this should be the default user behaviour for any cloud storage. Don't put unencrypted (company) data on a cloud infrastructure you don't have full control over.

Also reminds me of the (hyped?) "outrage" when a former facebook developer stated that they used to have a "default password" that allowed fb devs to log into every account and the media were like "omg they could have logged in and seen your photos". I mean... yeah they're the developers they could always do that even without the password

Re: AWS Support able to access any S3 object due to permission change

#56

Post by xvector »

L0g4n wrote: post #45

Such concerns specifically led to my decision of only uploading sensitive data to S3 with client side encryption. Since the aws cli tool only supports server side encryption with keys stored on amazon servers (where the non-default managed keys cost like 1 USD per month), I decided to simply symmetrically encrypt the backup of my syncthing data volume with AES256 using gnupg and only then pushing it to the S3 bucket.

Yeah, if you aren't using E2EE cloud storage you're doing it wrong.

Re: AWS Support able to access any S3 object due to permission change

#57

Post by time0ut »

StreamBright wrote: post #53
twosugars wrote: post #21

When I worked in aws, this is primarily used to check for permissions of an object. I know how dumb customers can be, for the most part this is used to see why a customer cannot delete a bucket or object those sort of things. I don't remember having ability to see actual customers data only metadata is accessible. Edit: Based on what I know, I'm pretty sure support will not be able see any of the customers data.

> I know how dumb customers can be Or how broken is the tooling for IAM + S3 + other services (for example Athena and Glue). Several times I had to explain to support that we do not want s3:* anywhere in our infra because they insisted that is the easiest solution so they do not need to waste their precious (paid by us) time on figuring out which exact permission is missing that I as a customer have no way of figurin…

This. The tooling and error messaging around IAM is inconsistent and lacking. I’ve even seen AWS support be completely wrong about why IAM is denying something, so I am guessing their internal tooling isn’t much better.

Re: AWS Support able to access any S3 object due to permission change

#59

Post by grumple »

twosugars wrote: post #25
trzmiel4 wrote: post #24

Earlier quoted context omitted.

> I know how dumb customers can be I find this insulting as a customer. Is AWS usually contemptuous of its customers? I don't think I've ever called my customer "dumb", and working as a consultant I've seen all kinds of interesting things. People make mistakes. They're always in a hurry. They may have a hard time understanding ambiguous, complex or incomplete documentation. The interface may be confusing and lead the…

You sound like never worked in support area.

I've had many AWS support engineers (and higher engineers) look at things in our env and say "I've never seen that before" and have no clue what was happening. It's a two way street. Everybody can't know everything. And remember that many devs in the real world have much broader domains than AWS engineers - I have to know every nuance about 30 AWS services, as well as my own applications and my own domain. An AWS engineer would be limited to having a deep understanding of one or a few services, and has internal experts on individual services to reach out to when they don't have some information. But sometimes even AWS devs might not be aware of a little line in the Lambda docs like "Background processes or callbacks that were initiated by your Lambda function and did not complete when the function ended resume if Lambda reuses the execution environment. Make sure that any background processes or callbacks in your code are complete before the code exits." [1] There are gotchas like this with every service, and missing a single line within the novella of docs AWS provides for each service is not a significant failing. There are also issues and concerns that are completely undocumented and are only learned with experience.

As a developer for a SaaS, I have to spend some time on support every day, including for devs who have refused to read our documentation for a particular service we provide (and the only one these devs use). It's frustrating, I know. You should assume that the developers who are your customers are unlikely to be stupid, and are instead just not informed about something or haven't read the docs (maybe they didn't know where to look, or like many, they are too busy to justify spending a day reading the docs for lambda). Best thing to do is direct them to the relevant parts of the documentation and do your best to help those people.

1. https://docs.aws.amazon.com/lambda/latest/dg/runtimes-contex...

Re: AWS Support able to access any S3 object due to permission change

#60

Post by chakerb »

L0g4n wrote: post #45

Such concerns specifically led to my decision of only uploading sensitive data to S3 with client side encryption. Since the aws cli tool only supports server side encryption with keys stored on amazon servers (where the non-default managed keys cost like 1 USD per month), I decided to simply symmetrically encrypt the backup of my syncthing data volume with AES256 using gnupg and only then pushing it to the S3 bucket.

Cloudstore[1] can do something similar automatically.

[1] https://github.com/infor-cloud/cloud-store

Post reply on HN