Live data from Hacker News

Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

theregister.com

31–40 of 162 posts

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#31
post #30
post #10

How did they track if the modification worked? By calling home? Because then they leaked PII

This is the Javascript SDK so checking any website which uses it would be enough to see if the modification is present.

Twilio says the only modification done to the script was to check if the modification could be done. If the hackers modified the script so it would call their own servers to check for that, they now have a list of IP addresses and timestamps.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#32

Aws should start taking blame for this shit, then maybe they would start auditing their customers setups...

AWS's shared responsibility model is clear. AWS is responsible for security of the cloud. The customer is responsible for the security in the cloud (i.e. the customer's resources). By the way, enterprise support customers do get access to well-architected reviews by AWS.

If you ask for help from AWS, AWS will provide it. It may not be free, but it's available.

Even if AWS were to start proactively auditing customer setups, how in the world is AWS supposed to know what a customer's usecase is? Nevermind the fact it's a breach of the customer's privacy to just go rooting around in the customer's account without permission.

But let's assume AWS is going to take responsibility for customers' configuration decisions and violate customers' privacy by proactively auditing their accounts. Would AWS auditing Twilio's configuration here work?

The default is for S3 buckets to be private. The customer has to take specific, affirmative steps to give s3 buckets public access. You really have to jump through hoops to make a bucket public accessible.

Since the Twilio chose to make their bucket public, AWS auditing Twilio's setup wouldn't be helpful. AWS would just assume the Twilio knows what they're doing. How is AWS supposed to know there is a misconfiguration? Because Twilio clearly decided to make their S3 bucket publicly available.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#33

S3 security is a pain. S3 options are tricky to master, docs are beyond confusing. A common S3 use case is: Use a Bucket for Read-only static content, (js, html, imgs). From a dev perspective ideally it will work like a protected folder + Web Server, whereas the webserver will have read-only privileges, but reality is far more complicated. I can understand why was easy for Twillio to have this infosec issue, since we…

Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account. If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL. Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a sin…

It is way too easy to screw up S3 bucket permissions. I caught it once when a coworker told me they setup a bucket for testing that allowed employees to upload, I checked the permissions and it allowed the world to upload.

It would be much harder make that mistake with Google Cloud Storage, at least in the web console. GCS also supports disabling bucket ACLs permanently at bucket creation time, and that is the option they recommend[1].

AWS does scan and alert for some public bucket issues. But with all the public S3 issues, I think it is clear there needs to be more prominent warnings in the AWS console, the command line, and with tools like Terraform.

[1] https://cloud.google.com/storage/docs/access-control

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#35

Having an S3 Bucket with writing permissions to anyone is a major fuck-up. I know many people open buckets to the world at the read-level for whatever reason. I do that myself for static websites. But also writing permissions? This requires a Bucket Policy that is written explicitly to allow this. I don't want to make assumptions because of course we don't know all the information and I personally know very little ab…

Yeah this is absolutely inexcusable for a public tech giant. I wonder how incompetently secured the rest of their backend is. This is terrifying, the amount of PII flowing through Twilio is insane, there really needs to be a way to bite companies that completely ignore all of these security compliance checks they say they pass...

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#36

S3 security is a pain. S3 options are tricky to master, docs are beyond confusing. A common S3 use case is: Use a Bucket for Read-only static content, (js, html, imgs). From a dev perspective ideally it will work like a protected folder + Web Server, whereas the webserver will have read-only privileges, but reality is far more complicated. I can understand why was easy for Twillio to have this infosec issue, since we…

Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account. If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL. Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a sin…

It is absolutely a pain. I'm using S3 for the first time on a project right now and literally every time I have to do anything in AWS I end up confused and scared that I'm leaving a wide open security vulnerability. The documentation is complete insanity to anyone just trying to accomplish what should be an extremely common use case in a reasonable amount of time.

Before using AWS I was in the 'what morons!' camp whenever some company got breached via leaving things wide open on AWS. Now I'm in the 'ok I understand why its so common' camp.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#37
post #21

Earlier quoted context omitted.

Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account. If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL. Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a sin…

This is the correct design pattern as of 2020. Roles + STS, and always be stingy when granting a role any permissions, both at the resource or API level. But, as others have said, the documentation at times is poor at best, and it can be easy to make mistakes. Some preventives every ops team can take are: 1. Never make manual permission changes in the console. Everything is done via SCM’d code. 2. All permission chan…

> Never make manual permission changes in the console. Everything is done via SCM’d code.

The first time I heard “infrastructure as code” I thought it was some dumb new buzz words, but no, it’s good advice. Your cloud formation, serverless, terraform whatever should be stored in GIT a stack replace is how changes should be made.

I consider the console as read-only.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#38
post #5

If it’s world-writable, then is it really a break-in?

"In local news, stranger off the street walks through residence's open door and moves around furniture."

Well, even in such a world, that would still be considered trespassing.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#39
post #21

Earlier quoted context omitted.

This is the correct design pattern as of 2020. Roles + STS, and always be stingy when granting a role any permissions, both at the resource or API level. But, as others have said, the documentation at times is poor at best, and it can be easy to make mistakes. Some preventives every ops team can take are: 1. Never make manual permission changes in the console. Everything is done via SCM’d code. 2. All permission chan…

> Never make manual permission changes in the console. Everything is done via SCM’d code. The first time I heard “infrastructure as code” I thought it was some dumb new buzz words, but no, it’s good advice. Your cloud formation, serverless, terraform whatever should be stored in GIT a stack replace is how changes should be made. I consider the console as read-only.

IMO this is code/console dichotomy is a design flaw in most cloud offerings that will haunt the industry for a long time.

The web console should be just another UI for editing the configuration in source control. As an example of why that's important, a UI can give you a warning and use affordances to guide beginners away from dangerous setups. In the case of S3, a slider for basic access level can use colors and show a list of objects that would become world readable or writable at a given level (for that matter why is world writable even an option?). A giant YAML file looks the same whether it's safe or dangerous.

If you really don't want to use a UI, then maybe abstracting a linting layer that feeds the exact same warnings to the UI as to your CI system and text editor would help.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#40

Earlier quoted context omitted.

Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account. If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL. Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a sin…

It is way too easy to screw up S3 bucket permissions. I caught it once when a coworker told me they setup a bucket for testing that allowed employees to upload, I checked the permissions and it allowed the world to upload. It would be much harder make that mistake with Google Cloud Storage, at least in the web console. GCS also supports disabling bucket ACLs permanently at bucket creation time, and that is the option…

>GCS also supports disabling bucket ACLs permanently at bucket creation time, and that is the option they recommend[1].

S3 does too. There's an entire page during the creation wizard that is dedicated to blocking any and all public access, even causing the bucket to ignore ACLs or other settings that would otherwise expose the bucket publicly. All public access is disabled by default, and enabling it actually requires the user to actively uncheck 5 different checkboxes, each of which explains that unchecking it will open the bucket up to public access, and then even requires an additional attestation, in a large orange warning box, that says you acknowledge that the options you chose will result in the bucket being public.

I'll be the first person to tell you that AWS is overcomplicated and hard to use, but this isn't that. IAM and bucket policies are a pain to work with, but if you screw up with those, the worse you will do is expose your bucket internally. But exposing a bucket publicly to the internet is an entirely different act, and there's not really any excuse for it other than just not reading the directions.

However, neither this option in S3 nor the option you linked in GCS would have served Twilio's use case. They wanted their bucket to be publicly accessible, just not publicly writable. That's an entirely different access management issue.

Post reply on HN