Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

theregister.com

Post reply on HN
Print view View original

111–120 of 162 posts

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#111

Post by stevehawk »

fimbulvetr wrote: post #103
kirillzubovsky wrote: post #65

Earlier quoted context omitted.

Potentially because it was spun 10 years ago while permissions were basic/different/even harder to track, and has since fallen off the wagon? It seems this incident has reminded Twilio to audit all their access right, which is a net positive. Actually, imagine you were an employee who wanted to get this done, but couldn't get management approval? This is one way to fix things ... :)

For such an important bucket, they should have a. adopted the resource into a Cloudformation stack & a1. Enabled drift detection. b. Use an AWS config rule to monitor (appropriate) s3 buckets for any public access.

1. Drift detection is a relatively new feature. 2. CloudFormation was so basic for so long that most AWS professionals don't use it. They use Terraform or a similar, non AWS product.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#112

Post by ficklepickle »

rumanator wrote: post #94
kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

> They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. In AWS you need to explicitly set a bucket as publicly accessible, which requires you to go through a couple of dialogs where you're expected to say "yes, yes I want to allow others to access this S3 bucket." Afterwards you still get a bucket that does not have write access…

This. It is actually quite difficult to make a bucket writable by anyone. I don't understand how it can be done accidentally.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#113

Post by heavenlyblue »

ryanlol wrote: post #107
dsl wrote: post #100

Earlier quoted context omitted.

Magecart is a well known framework in the underground for stealing credit card numbers from websites (if they can get the JavaScript running on a checkout page). If you hit it directly they will redirect to a random "Spin the wheel win a prize" type site in an attempt to hide the malicious nature of the payload.

This is not magecart.

So they got really lucky it wasn’t magecart - does this change anything?

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#114

Post by morvita »

orf wrote: post #110
bjclark wrote: post #87

Earlier quoted context omitted.

The flaw in your logic is that "S3 is front-and-center the source of your business" is never true with AWS. To do anything in AWS there's at least 3 or 4 vaguely connected services, including IAM and RAM as a completely separate UIs. Usually with S3 you also have Cloudfront or some other CDN. You probably also have Cloudwatch logs and CLoudtrail event tracking. You might have VPCs involved. Some or all of these thing…

Each of these services are highly different concerns and don’t belong together, and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform. If you’ve got one team managing your logging who don’t talk to whatever team who manages your s3/cloudfront setup, then that’s your problem.

> and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform.

I love Terraform, but the fact that you require an entirely different company's orchestration tools to make AWS "not that hard to orchestrate" is evidence of how poor the UX of AWS is.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#115

Post by ficklepickle »

oefrha wrote: post #105
aka1234 wrote: post #102

Earlier quoted context omitted.

Thanks for posting this. I'm really impressed with the transparency Twilio showed in actually admitting to having such a silly, silly bucket policy. Not impressed that it was there in the first place; but that should go without saying. This incident report should really put to bed all of the "It's AWS's fault for making things so complex" complaints. (To be clear, it won't... but it should.) Even a cursory look at th…

Really not impressed with the obligatory "really impressed with transparency" pat-on-the-back under every incident report for a big corp screw-up that provides any details at all. And stating to the press the clearly malicious payload is "non-malicious" (assuming TFA didn't lie about Twilio's statement)? That's ridiculous.

Even if the payload was not malicious when they looked, it could change at any time. I don't see how that can be confidently labeled non-malicious.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#117

Post by zimbatm »

kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

Things are much better than what they used to be.

AWS Config is easy to setup and can be used to warn for that type of access-control issues. It's a good layer of additional defense.

The new S3 UI is also much better and won't let you do insecure things by default.

AWS also sends emails when they find an open bucket. I don't know exactly when those are triggered but I got them in the past.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#118

Post by aldarisbm »

bjclark wrote: post #87
ericbarrett wrote: post #85

Earlier quoted context omitted.

I really have to disagree here, at least with your conclusions re complexity. No doubt AWS S3 configuration is very technical and has a lot of docs and corner cases. However, if you are a company like Twilio, and S3 is front-and-center the source of your business (and if it's serving your SDK, it probably is), your S3 bucket permissions are clearly worthy of periodic review. And you should have this done by somebody…

The flaw in your logic is that "S3 is front-and-center the source of your business" is never true with AWS. To do anything in AWS there's at least 3 or 4 vaguely connected services, including IAM and RAM as a completely separate UIs. Usually with S3 you also have Cloudfront or some other CDN. You probably also have Cloudwatch logs and CLoudtrail event tracking. You might have VPCs involved. Some or all of these thing…

Might be a war-crime, but people at the enterprise level like twilio, don't do anything at the UI level, and if someone was to do something at the UI level, AWS blocks public access _by default_

AWS console was only complicated as a complete noob. Once you understand how services work you actually realize is decent.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#119

Post by aldarisbm »

morvita wrote: post #114
orf wrote: post #110

Earlier quoted context omitted.

Each of these services are highly different concerns and don’t belong together, and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform. If you’ve got one team managing your logging who don’t talk to whatever team who manages your s3/cloudfront setup, then that’s your problem.

> and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform. I love Terraform, but the fact that you require an entirely different company's orchestration tools to make AWS "not that hard to orchestrate" is evidence of how poor the UX of AWS is.

But terraform is using the aws sdk behind the covers.

Most of the stuff available with terraform is available with CloudFormation minus a thing or two. (takes them a minute to catch-up to other teams new services/features)

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#120

Post by aldarisbm »

nathas wrote: post #104
kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

There's a new tool around that helps these cases be prevented at an organization level called CfnGuard: https://aws.amazon.com/about-aws/whats-new/2020/06/introduci... Obviously you have to be using CloudFormation, so if they're using i.e. Terraform they're out of luck.

Hashicorp sentinel
Post reply on HN