Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

theregister.com

Post reply on HN
Print view View original

101–110 of 162 posts

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#101

Post by harrisonjackson »

rafaelturk wrote: post #7

S3 security is a pain. S3 options are tricky to master, docs are beyond confusing. A common S3 use case is: Use a Bucket for Read-only static content, (js, html, imgs). From a dev perspective ideally it will work like a protected folder + Web Server, whereas the webserver will have read-only privileges, but reality is far more complicated. I can understand why was easy for Twillio to have this infosec issue, since we…

Agreed. Misconfigured s3 buckets have to be in the top 1% of of causes for data leaks, hacking, etc security issues in the past decade.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#102

Post by aka1234 »

kolencherry wrote: post #93

Hey folks, we've published a post with more details on the incident here: https://www.twilio.com/blog/incident-report-taskrouter-js-sd... (I work for Twilio)

Thanks for posting this. I'm really impressed with the transparency Twilio showed in actually admitting to having such a silly, silly bucket policy. Not impressed that it was there in the first place; but that should go without saying.

This incident report should really put to bed all of the "It's AWS's fault for making things so complex" complaints. (To be clear, it won't... but it should.)

Even a cursory look at that bucket policy should tell you something named "Allow Public Read" should NOT be associated with anything named 'Put'. This takes 0 AWS knowledge to figure out.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#103

Post by fimbulvetr »

kirillzubovsky wrote: post #65
whoisjuan wrote: post #28

Having an S3 Bucket with writing permissions to anyone is a major fuck-up. I know many people open buckets to the world at the read-level for whatever reason. I do that myself for static websites. But also writing permissions? This requires a Bucket Policy that is written explicitly to allow this. I don't want to make assumptions because of course we don't know all the information and I personally know very little ab…

Potentially because it was spun 10 years ago while permissions were basic/different/even harder to track, and has since fallen off the wagon? It seems this incident has reminded Twilio to audit all their access right, which is a net positive. Actually, imagine you were an employee who wanted to get this done, but couldn't get management approval? This is one way to fix things ... :)

For such an important bucket, they should have

a. adopted the resource into a Cloudformation stack & a1. Enabled drift detection.

b. Use an AWS config rule to monitor (appropriate) s3 buckets for any public access.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#104

Post by nathas »

kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

There's a new tool around that helps these cases be prevented at an organization level called CfnGuard: https://aws.amazon.com/about-aws/whats-new/2020/06/introduci...

Obviously you have to be using CloudFormation, so if they're using i.e. Terraform they're out of luck.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#105

Post by oefrha »

aka1234 wrote: post #102
kolencherry wrote: post #93

Hey folks, we've published a post with more details on the incident here: https://www.twilio.com/blog/incident-report-taskrouter-js-sd... (I work for Twilio)

Thanks for posting this. I'm really impressed with the transparency Twilio showed in actually admitting to having such a silly, silly bucket policy. Not impressed that it was there in the first place; but that should go without saying. This incident report should really put to bed all of the "It's AWS's fault for making things so complex" complaints. (To be clear, it won't... but it should.) Even a cursory look at th…

Really not impressed with the obligatory "really impressed with transparency" pat-on-the-back under every incident report for a big corp screw-up that provides any details at all.

And stating to the press the clearly malicious payload is "non-malicious" (assuming TFA didn't lie about Twilio's statement)? That's ridiculous.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#106

Post by untog »

Someone1234 wrote: post #97
untog wrote: post #26

Earlier quoted context omitted.

It's difficult. I like the idea of subresource integrity a lot, but a counter example: Twilio discovers a bug in their JS SDK. Without subresource integrity they can push up a fix and propagate it to all clients immediately. With subresource integrity they would need every client to update their web site to change the integrity attribute. If it was a serious bug then that could be a huge issue. Even putting the secur…

> Without subresource integrity they can push up a fix and propagate it to all clients immediately. In reality, no you cannot. You have multiple layers of cache between the S3 bucket and rendering, unless you disable caching entirely at a massive increased cost. Some of these caches are poorly behaving (e.g. intermediary caching). The correct way of doing this ALREADY is to increment the version number in the URL (e.…

You do not have absolute certainty that every old cached version of a file you have published has been purged by every third party cache, no. But the vast, _vast_ majority do get purged very quickly. I've done this myself when in a bind many times and it's worked fine (and I've been able to verify that via error logging).

> There's a reason why "sdk-latest.js" is largely dead concept from a bygone era

Is it, though? From a quick check, it's what Google Maps does. It's what the Facebook SDK does. We already know it's what Twilio does.

> This argument is essentially: "I'm doing it wrong, and Subresource Integrity would stop that, so it is a non-starter."

Zero dispute with that characterisation from me. But it's how things already operate in the real world. I'll join you in shouting from the rooftops that people shouldn't be doing it, but that doesn't really get you any closer to actually stopping them. For a great many people the flexibility to quickly push up changes is a feature, not a bug.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#107

Post by ryanlol »

dsl wrote: post #100
FDSGSG wrote: post #83

Earlier quoted context omitted.

FWIW, that's a redirector and not a skimmer.

Magecart is a well known framework in the underground for stealing credit card numbers from websites (if they can get the JavaScript running on a checkout page). If you hit it directly they will redirect to a random "Spin the wheel win a prize" type site in an attempt to hide the malicious nature of the payload.

This is not magecart.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#109

Post by aaronbrethorst »

ericbarrett wrote: post #85
kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

I really have to disagree here, at least with your conclusions re complexity. No doubt AWS S3 configuration is very technical and has a lot of docs and corner cases. However, if you are a company like Twilio, and S3 is front-and-center the source of your business (and if it's serving your SDK, it probably is), your S3 bucket permissions are clearly worthy of periodic review. And you should have this done by somebody…

your S3 bucket permissions are clearly worthy of periodic review.

better tooling makes this easier to do.

And you should have this done by somebody who knows it well, or who learns it well in the process.

No one knows S3 better than the AWS S3 team.

It took us half a day to get the bucket locked down to where only the root AWS IAM user—secured by physical 2fac—was permitted to change the SDK

Or a small feature team could spend a month or two improving the user experience so that literally every S3 customer doesn't have to have the technical acumen, knowledge of risks, and time to get this right. 0.5 days * a million users (to be conservative) is 500,000 days.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#110

Post by orf »

bjclark wrote: post #87
ericbarrett wrote: post #85

Earlier quoted context omitted.

I really have to disagree here, at least with your conclusions re complexity. No doubt AWS S3 configuration is very technical and has a lot of docs and corner cases. However, if you are a company like Twilio, and S3 is front-and-center the source of your business (and if it's serving your SDK, it probably is), your S3 bucket permissions are clearly worthy of periodic review. And you should have this done by somebody…

The flaw in your logic is that "S3 is front-and-center the source of your business" is never true with AWS. To do anything in AWS there's at least 3 or 4 vaguely connected services, including IAM and RAM as a completely separate UIs. Usually with S3 you also have Cloudfront or some other CDN. You probably also have Cloudwatch logs and CLoudtrail event tracking. You might have VPCs involved. Some or all of these thing…

Each of these services are highly different concerns and don’t belong together, and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform.

If you’ve got one team managing your logging who don’t talk to whatever team who manages your s3/cloudfront setup, then that’s your problem.

Post reply on HN