How did they track if the modification worked? By calling home? Because then they leaked PII
This is the Javascript SDK so checking any website which uses it would be enough to see if the modification is present.
Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK
31–40 of 162 posts
Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK
#32Aws should start taking blame for this shit, then maybe they would start auditing their customers setups...
If you ask for help from AWS, AWS will provide it. It may not be free, but it's available.
Even if AWS were to start proactively auditing customer setups, how in the world is AWS supposed to know what a customer's usecase is? Nevermind the fact it's a breach of the customer's privacy to just go rooting around in the customer's account without permission.
But let's assume AWS is going to take responsibility for customers' configuration decisions and violate customers' privacy by proactively auditing their accounts. Would AWS auditing Twilio's configuration here work?
The default is for S3 buckets to be private. The customer has to take specific, affirmative steps to give s3 buckets public access. You really have to jump through hoops to make a bucket public accessible.
Since the Twilio chose to make their bucket public, AWS auditing Twilio's setup wouldn't be helpful. AWS would just assume the Twilio knows what they're doing. How is AWS supposed to know there is a misconfiguration? Because Twilio clearly decided to make their S3 bucket publicly available.
Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK
#33S3 security is a pain. S3 options are tricky to master, docs are beyond confusing. A common S3 use case is: Use a Bucket for Read-only static content, (js, html, imgs). From a dev perspective ideally it will work like a protected folder + Web Server, whereas the webserver will have read-only privileges, but reality is far more complicated. I can understand why was easy for Twillio to have this infosec issue, since we…
Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account. If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL. Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a sin…
It would be much harder make that mistake with Google Cloud Storage, at least in the web console. GCS also supports disabling bucket ACLs permanently at bucket creation time, and that is the option they recommend[1].
AWS does scan and alert for some public bucket issues. But with all the public S3 issues, I think it is clear there needs to be more prominent warnings in the AWS console, the command line, and with tools like Terraform.
Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK
#34If it’s world-writable, then is it really a break-in?
Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK
#35Having an S3 Bucket with writing permissions to anyone is a major fuck-up. I know many people open buckets to the world at the read-level for whatever reason. I do that myself for static websites. But also writing permissions? This requires a Bucket Policy that is written explicitly to allow this. I don't want to make assumptions because of course we don't know all the information and I personally know very little ab…
Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK
#36S3 security is a pain. S3 options are tricky to master, docs are beyond confusing. A common S3 use case is: Use a Bucket for Read-only static content, (js, html, imgs). From a dev perspective ideally it will work like a protected folder + Web Server, whereas the webserver will have read-only privileges, but reality is far more complicated. I can understand why was easy for Twillio to have this infosec issue, since we…
Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account. If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL. Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a sin…
Before using AWS I was in the 'what morons!' camp whenever some company got breached via leaving things wide open on AWS. Now I'm in the 'ok I understand why its so common' camp.
Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK
#37Earlier quoted context omitted.
Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account. If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL. Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a sin…
This is the correct design pattern as of 2020. Roles + STS, and always be stingy when granting a role any permissions, both at the resource or API level. But, as others have said, the documentation at times is poor at best, and it can be easy to make mistakes. Some preventives every ops team can take are: 1. Never make manual permission changes in the console. Everything is done via SCM’d code. 2. All permission chan…
The first time I heard “infrastructure as code” I thought it was some dumb new buzz words, but no, it’s good advice. Your cloud formation, serverless, terraform whatever should be stored in GIT a stack replace is how changes should be made.
I consider the console as read-only.
Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK
#38Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK
#39Earlier quoted context omitted.
This is the correct design pattern as of 2020. Roles + STS, and always be stingy when granting a role any permissions, both at the resource or API level. But, as others have said, the documentation at times is poor at best, and it can be easy to make mistakes. Some preventives every ops team can take are: 1. Never make manual permission changes in the console. Everything is done via SCM’d code. 2. All permission chan…
> Never make manual permission changes in the console. Everything is done via SCM’d code. The first time I heard “infrastructure as code” I thought it was some dumb new buzz words, but no, it’s good advice. Your cloud formation, serverless, terraform whatever should be stored in GIT a stack replace is how changes should be made. I consider the console as read-only.
The web console should be just another UI for editing the configuration in source control. As an example of why that's important, a UI can give you a warning and use affordances to guide beginners away from dangerous setups. In the case of S3, a slider for basic access level can use colors and show a list of objects that would become world readable or writable at a given level (for that matter why is world writable even an option?). A giant YAML file looks the same whether it's safe or dangerous.
If you really don't want to use a UI, then maybe abstracting a linting layer that feeds the exact same warnings to the UI as to your CI system and text editor would help.
Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK
#40Earlier quoted context omitted.
Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account. If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL. Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a sin…
It is way too easy to screw up S3 bucket permissions. I caught it once when a coworker told me they setup a bucket for testing that allowed employees to upload, I checked the permissions and it allowed the world to upload. It would be much harder make that mistake with Google Cloud Storage, at least in the web console. GCS also supports disabling bucket ACLs permanently at bucket creation time, and that is the option…
S3 does too. There's an entire page during the creation wizard that is dedicated to blocking any and all public access, even causing the bucket to ignore ACLs or other settings that would otherwise expose the bucket publicly. All public access is disabled by default, and enabling it actually requires the user to actively uncheck 5 different checkboxes, each of which explains that unchecking it will open the bucket up to public access, and then even requires an additional attestation, in a large orange warning box, that says you acknowledge that the options you chose will result in the bucket being public.
I'll be the first person to tell you that AWS is overcomplicated and hard to use, but this isn't that. IAM and bucket policies are a pain to work with, but if you screw up with those, the worse you will do is expose your bucket internally. But exposing a bucket publicly to the internet is an entirely different act, and there's not really any excuse for it other than just not reading the directions.
However, neither this option in S3 nor the option you linked in GCS would have served Twilio's use case. They wanted their bucket to be publicly accessible, just not publicly writable. That's an entirely different access management issue.