I’m honestly surprised no one has mentioned log4j in this thread yet. What if this change was part of an outside attack?
That's an excellent point, especially since AWS has a big Java presence.
AWS Support able to access any S3 object due to permission change
91–100 of 134 posts
Re: AWS Support able to access any S3 object due to permission change
#92I'm not sure what the hoo hah is about. All that's changed is that this role is now visible in IAMS and any API calls by Amazon support tools will be logged in AWS CloudTrail - I don't think AWS have any more access now than they did before. It's obvious to anyone who has had problems with any AWS services (lambda functions that couldn't be deleted, non properly propagating/operational services) that support has acce…
The policy in question had “s3:GetObject” permission to “*” added for a few hours. And CloudTrail logging doesn’t capture GetObject API requests by default. This role is specifically for metadata only access. Yes, service teams have behind-the-scenes escalation tools for bugs on the backend, but the more numerous front-line support staff should only be able to view metadata. The GetObject permission would have let th…
You really shouldn't be storing "potentially sensitive data" in a bucket unencrypted, though.
Re: AWS Support able to access any S3 object due to permission change
#93So its completely Amazon's storage then? Given that and it's not your computer, so it also not your storage (even it is encrypted)? If all of the above is true, is it time to panic? Perhaps maybe that why the cloud is somewhat of a scam anyways?
I do wonder if the push for cloud native development is just an inbound sales technique by Amazon/Microsoft/Google
Re: AWS Support able to access any S3 object due to permission change
#94I guess taking some risks as I am not really authorized to use social media, but at AWS there is a sauron like focus on not letting internal engineers view customer data. On my service we don't even have tools to do it and the security controls are very tight and getting tighter all the time. It's a big fucking deal, if not the biggest fucking deal, besides KTLO. Posting to get ahead of some of the comment here on se…
While I recognize that's the ideal goal, and most appropriate, I think there may be gaps in practice. My team just discovered yesterday that a Kinesis Firehose Stream can still deliver to an S3 bucket that lacks a bucket policy and whose ACLs disable any access to it at all. That diminished my confidence a little bit that all teams are perfectly in compliance of the overall goal.
Re: AWS Support able to access any S3 object due to permission change
#95Such concerns specifically led to my decision of only uploading sensitive data to S3 with client side encryption. Since the aws cli tool only supports server side encryption with keys stored on amazon servers (where the non-default managed keys cost like 1 USD per month), I decided to simply symmetrically encrypt the backup of my syncthing data volume with AES256 using gnupg and only then pushing it to the S3 bucket.
Is there a managed service that does this? A Dropbox-like UX that client-side encrypts everything and then persists it on S3 or the like. I don’t want to fiddle with it myself but I’d pay for that kind of service
Re: AWS Support able to access any S3 object due to permission change
#96Such concerns specifically led to my decision of only uploading sensitive data to S3 with client side encryption. Since the aws cli tool only supports server side encryption with keys stored on amazon servers (where the non-default managed keys cost like 1 USD per month), I decided to simply symmetrically encrypt the backup of my syncthing data volume with AES256 using gnupg and only then pushing it to the S3 bucket.
Is there a managed service that does this? A Dropbox-like UX that client-side encrypts everything and then persists it on S3 or the like. I don’t want to fiddle with it myself but I’d pay for that kind of service
rclone can do it on the fly (no need to store the encrypted version on upload or download) which simplifies the process
Re: AWS Support able to access any S3 object due to permission change
#97Okay, opinions up front: I don't think this is worthy of "declaring a security incident. Having some experience working behind the scenes, just because this policy was changes this way doesn't mean "All AWS Support personnel had unrequited access to your S3 objects." To me, this reads as Twitter inflammatory nonsense. Here's why: * KMS Encrypted objects would not be accessible because the support personnel would need…
"Declaring an incident" means there's something to investigate, it didn't mean anything bad has happened: it's detection of a non-conformity. The output of the incident would look similar to what you wrote. Any time the wrong permissions are assigned and confidentiality is potentially breached, I think you have to have an incident. Arguably in some jurisdictions, it's a legal requirement to ensure you have a near mis…
Exactly. This is something we'll be reviewing in our monthly security review at work, discussing what the impact was, why we were not impacted, and any action items we want to take.
Declaring an incident doesn't mean sending out a breach report or anything particularly dramatic, though I can see how as an outsider it may sound that way.
Re: AWS Support able to access any S3 object due to permission change
#98I’m honestly surprised no one has mentioned log4j in this thread yet. What if this change was part of an outside attack?
There's no reason to believe that to be the case.
Re: AWS Support able to access any S3 object due to permission change
#99Earlier quoted context omitted.
The policy in question had “s3:GetObject” permission to “*” added for a few hours. And CloudTrail logging doesn’t capture GetObject API requests by default. This role is specifically for metadata only access. Yes, service teams have behind-the-scenes escalation tools for bugs on the backend, but the more numerous front-line support staff should only be able to view metadata. The GetObject permission would have let th…
You really shouldn't be storing "potentially sensitive data" in a bucket unencrypted, though.
The problem is nobody who is getting upset about this had any reasonable expectations to begin with. They're just now realizing they might need to protect their data against attackers inside AWS, and now they're caught with their pants down, so cue the hand-wringing.
Re: AWS Support able to access any S3 object due to permission change
#100Earlier quoted context omitted.
> I know how dumb customers can be Maybe a more constructive way to look at this would be that people simply do "dumb" things. In customer support where you only see those moments, it might not always seem that way, but dealing with people's simple mistakes is also educating them to do better next time.
People can be ignorant, lazy, not give a shit about the work they are doing, have poor learning ability and or skills, and cross their fingers, mashing buttons, hoping everything just works, and then expect everyone else around them to help them out of their screw ups. If you've ever worked in CS, or known anyone that works in CS, you know that there are an absolute fucking shitload of these people. Often in roles th…
That's true. It's also true that AWS's IAM system is pretty complex and not incredibly well designed. AWS internally makes mistakes with it with some regularity.