Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK • Hacker News Board

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#81

Post by marcocampos » Wed, Jul 22, 2020, 5:07 PM UTC

MrBuddyCasino wrote: post #76

kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

I am still amazed some people want to do devops. Write yaml, run script, wait 15 minutes, oh no a typo lets retry. The developer experience is worse than nearly everything that came before it, and I would include ancient PHP and callback-hell nodejs in that. The AWS web UI isn’t much better. Looking forward to finally try Pulumi in a project.

How will Pulumi fix the issues you mentioned?

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#82

Post by Osiris » Wed, Jul 22, 2020, 5:07 PM UTC

kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

I used Azure for a year on a project and they did seem to have a pretty big focus on UI/UX. Without having previously done any cloud management work directly, I found it fairly straightforward.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#83

Post by FDSGSG » Wed, Jul 22, 2020, 5:08 PM UTC

oefrha wrote: post #74

tyingq wrote: post #50

Doesn't sound so non-malicious to me: "Specifically, the modification added code to the end of the TaskRouter.js v1.20 SDK that made an HTTP GET request to hxxps://gold.platinumus.top/track/awswrite?q=dmn and followed the URL returned in the HTML by that request."

> followed the URL returned in the HTML by that request Translated: we can't possibly know it's non-malicious. TFA even has this to say: > And judging from the URL involved, it appears to be an attempt to install a payment-card skimmer – RiskIQ has spotted the same URL in other S3 buckets targeted by miscreants. Details in the linked blog post[0]. So, very much malicious. Why the hell they included "non-malicious" in…

FWIW, that's a redirector and not a skimmer.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#84

Post by meritt » Wed, Jul 22, 2020, 5:08 PM UTC

ed25519FUUU wrote: post #8

rafaelturk wrote: post #7

S3 security is a pain. S3 options are tricky to master, docs are beyond confusing. A common S3 use case is: Use a Bucket for Read-only static content, (js, html, imgs). From a dev perspective ideally it will work like a protected folder + Web Server, whereas the webserver will have read-only privileges, but reality is far more complicated. I can understand why was easy for Twillio to have this infosec issue, since we…

Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account. If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL. Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a sin…

I have external partners who provide their own principal and I provision a subset of bucket access at the bucket policy level. Is there a better way to do this that's more manageable? The end users are using software or python code where they need to gain access via [access, secret, bucket, prefix/key].

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#85

Post by ericbarrett » Wed, Jul 22, 2020, 5:09 PM UTC

kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

I really have to disagree here, at least with your conclusions re complexity. No doubt AWS S3 configuration is very technical and has a lot of docs and corner cases. However, if you are a company like Twilio, and S3 is front-and-center the source of your business (and if it's serving your SDK, it probably is), your S3 bucket permissions are clearly worthy of periodic review. And you should have this done by somebody who knows it well, or who learns it well in the process.

I worked for a company where our CTO identified the same thing (unauthorized modification of JS SDK served via S3) as a potential vulnerability. It took us half a day to get the bucket locked down to where only the root AWS IAM user—secured by physical 2fac—was permitted to change the SDK, and then only after "unlocking" the configuration changes. Since the SDK was not updated frequently, having a little manual ceremony was fine. If it had been a high-touch path we would have had to do a little more engineering to keep it safe, but not that much more.

That said, I don't blame Twilio for not catching this; shit happens at a company with years of legacy built up. I do blame anybody who says "S3 is hard" and throws up their hands. You're a professional! Read the docs, they're dry but quite extensive! Play around with it in a test bucket! It's no harder than learning any other moderately complicated professional system. If you can learn Rust, you can learn S3's API. If you can learn Kubernetes, you can learn S3 ACLs and the IAM authz model. It may not be as interesting as the topics you want to read about, but it will probably be even more useful.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#86

Post by somethingwitty1 » Wed, Jul 22, 2020, 5:12 PM UTC

kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

I agree AWS can do a lot better with the tooling UX, but adding write permissions like this is a conscious decision. In this case, someone purposefully added it (outside the creation UX flow). I'm not sure we can put the blame squarely on AWS for this one.

And they definitely haven't been doing nothing. They've drastically redesigned the permissions UX around S3 over the years. They introduced GuardDuty to allow automatic threat detection (such as S3 bucket having open permissions) which can be applied across your organization (not just an account). They definitely have to keep doing more, but implying they are doing nothing just isn't true.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#87

Post by bjclark » Wed, Jul 22, 2020, 5:31 PM UTC

ericbarrett wrote: post #85

kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

I really have to disagree here, at least with your conclusions re complexity. No doubt AWS S3 configuration is very technical and has a lot of docs and corner cases. However, if you are a company like Twilio, and S3 is front-and-center the source of your business (and if it's serving your SDK, it probably is), your S3 bucket permissions are clearly worthy of periodic review. And you should have this done by somebody…

The flaw in your logic is that "S3 is front-and-center the source of your business" is never true with AWS.

To do anything in AWS there's at least 3 or 4 vaguely connected services, including IAM and RAM as a completely separate UIs. Usually with S3 you also have Cloudfront or some other CDN. You probably also have Cloudwatch logs and CLoudtrail event tracking. You might have VPCs involved. Some or all of these things, in an org like Twillio, could have entirely different teams managing them.

The problem isn't that S3 is hard. The problem is, as OP suggests, the UX of actually doing anything non-trivial across half a dozen services is somewhere between abysmal and war-crime.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#88

Post by donor20 » Wed, Jul 22, 2020, 5:32 PM UTC

aka1234 wrote: post #32

coding123 wrote: post #12

Aws should start taking blame for this shit, then maybe they would start auditing their customers setups...

AWS's shared responsibility model is clear. AWS is responsible for security of the cloud. The customer is responsible for the security in the cloud (i.e. the customer's resources). By the way, enterprise support customers do get access to well-architected reviews by AWS. If you ask for help from AWS, AWS will provide it. It may not be free, but it's available. Even if AWS were to start proactively auditing customer s…

Not only do you have to jump through hoops to make it public, if you are worried can't you use Amazon Macie and have a separate org level view that alerts you to any public buckets?

https://aws.amazon.com/macie

I don't bother because it seems pretty clear what buckets are public.

That said, quick tip to make your life easier.

DON'T use S3 ACL's DON'T use S3 policies.

If I was AWS and didn't have so many customers I'd probably just create one mental model (IAM policies probably) as the place to manage things and block the rest.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#89

Post by Someone1234 » Wed, Jul 22, 2020, 5:34 PM UTC

Twilio's examples[0] for taskrouter.min.js, even now, don't include Subresource Integrity hashes. Subresource Integrity would have stopped the modified script from being loaded, and removed a single point of failure.

Subresource Integrity is supported by every evergreen browser and 80%+ of market share (no IE11/pre-Chromium Edge).

[0] https://www.twilio.com/docs/taskrouter/js-sdk/workspace/work...

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#90

Post by donor20 » Wed, Jul 22, 2020, 5:34 PM UTC

One thing I wish AWS would do.

Allow admins to "down feature" the account. Ie, simple mode.

Basically, allow admins to toggle OFF S3 ACL's and even S3 Bucket Policies - zero those out. Then let users use IAM policies for access which may be a more familiar / more used mental model.