Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

theregister.com

Post reply on HN
Print view View original

141–150 of 162 posts

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#141

Post by rumanator »

kirillzubovsky wrote: post #65
whoisjuan wrote: post #28

Having an S3 Bucket with writing permissions to anyone is a major fuck-up. I know many people open buckets to the world at the read-level for whatever reason. I do that myself for static websites. But also writing permissions? This requires a Bucket Policy that is written explicitly to allow this. I don't want to make assumptions because of course we don't know all the information and I personally know very little ab…

Potentially because it was spun 10 years ago while permissions were basic/different/even harder to track, and has since fallen off the wagon? It seems this incident has reminded Twilio to audit all their access right, which is a net positive. Actually, imagine you were an employee who wanted to get this done, but couldn't get management approval? This is one way to fix things ... :)

> because it was spun 10 years ago while permissions were basic/different/even harder to track, and has since fallen off the wagon?

That's not a AWS problem. That's a "our team failed to manage the service" problem.

This includes monitoring a publicly accessible bucket.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#142

Post by ryanlol »

heavenlyblue wrote: post #136
ryanlol wrote: post #116

Earlier quoted context omitted.

Of course it does.

If someone drove drunk and drove into a pedestrian who survives after a month in ER does this diminish the severity of drinking or not?

An incident where the pedestrian isn’t killed is almost certainly less severe than one where the pedestrian is killed.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#143

Post by bostonvaulter2 »

rumanator wrote: post #139
earthboundkid wrote: post #132

Earlier quoted context omitted.

Amazon probably can't change S3 for backwards compatibility reasons (there was an outcry about just changing some legacy URLs last year), but they definitely can and should make an "S4" which would be an S3 compatible storage layer with sane defaults. Essentially, people only ever want one of two things: - A private FTP bucket - A public static asset website (with built-in CDN and HTTPS!) Thing people never want: - A…

>Thing people never want: > >- A public upload bucket Thus is outright wrong. Public upload s3 buckets are AWS's go-to basic technique to handle file uploads. AWS's API Gateway has a max file limit, and AWS advises developers to leverage S3 and lambda triggers to implement that usecases.

Yes, but the upload url's should be signed by the server in that case.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#144

Post by kjeetgill »

ljoshua wrote: post #38
TheSpiciestDev wrote: post #34

Earlier quoted context omitted.

"In local news, stranger off the street walks through residence's open door and moves around furniture."

Well, even in such a world, that would still be considered trespassing.

Sure, but would you want to hear that from your bank? Who needs vaults when you can declare trespassing.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#145

Post by syntheticcdo »

bjclark wrote: post #87
ericbarrett wrote: post #85

Earlier quoted context omitted.

I really have to disagree here, at least with your conclusions re complexity. No doubt AWS S3 configuration is very technical and has a lot of docs and corner cases. However, if you are a company like Twilio, and S3 is front-and-center the source of your business (and if it's serving your SDK, it probably is), your S3 bucket permissions are clearly worthy of periodic review. And you should have this done by somebody…

The flaw in your logic is that "S3 is front-and-center the source of your business" is never true with AWS. To do anything in AWS there's at least 3 or 4 vaguely connected services, including IAM and RAM as a completely separate UIs. Usually with S3 you also have Cloudfront or some other CDN. You probably also have Cloudwatch logs and CLoudtrail event tracking. You might have VPCs involved. Some or all of these thing…

If you have CloudFront in front of your bucket, the bucket shouldn't even be world-readable, let alone world-writable. Permission should be delegated to CF to read out of the bucket.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#147

Post by john-shaffer »

rumanator wrote: post #139
earthboundkid wrote: post #132

Earlier quoted context omitted.

Amazon probably can't change S3 for backwards compatibility reasons (there was an outcry about just changing some legacy URLs last year), but they definitely can and should make an "S4" which would be an S3 compatible storage layer with sane defaults. Essentially, people only ever want one of two things: - A private FTP bucket - A public static asset website (with built-in CDN and HTTPS!) Thing people never want: - A…

>Thing people never want: > >- A public upload bucket Thus is outright wrong. Public upload s3 buckets are AWS's go-to basic technique to handle file uploads. AWS's API Gateway has a max file limit, and AWS advises developers to leverage S3 and lambda triggers to implement that usecases.

The bucket should never be publicly writable. The "go-to" techniques are signed POST requests [1] and signed URLs [2].

[1] https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-post-e...

[2] https://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrl...

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#148

Post by Corrado »

ficklepickle wrote: post #112
rumanator wrote: post #94

Earlier quoted context omitted.

> They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. In AWS you need to explicitly set a bucket as publicly accessible, which requires you to go through a couple of dialogs where you're expected to say "yes, yes I want to allow others to access this S3 bucket." Afterwards you still get a bucket that does not have write access…

This. It is actually quite difficult to make a bucket writable by anyone. I don't understand how it can be done accidentally.

In the past, S3 buckets were created with much less sane defaults and you didn't have the ability to unilaterally forbid any public access that you do today. Also, configuring Bucket Profiles is a confusing and hard to test situation. And if that weren't enough, S3 buckets also have ACLs (though they are deprecated) that represent yet another way to configure access to your buckets.

So, from the confusing UX to the multiple permissions settings, I can easily see how a company could get in this situation. In fact, the problem is so bad that AWS has revised the whole S3 thing several times, with each new iteration more secure than the last. Unfortunately, that doesn't help the company that configured their buckets 5 years ago and haven't touched them since.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#149

Post by jen20 »

boarnoah wrote: post #127
MrBuddyCasino wrote: post #76

Earlier quoted context omitted.

I am still amazed some people want to do devops. Write yaml, run script, wait 15 minutes, oh no a typo lets retry. The developer experience is worse than nearly everything that came before it, and I would include ancient PHP and callback-hell nodejs in that. The AWS web UI isn’t much better. Looking forward to finally try Pulumi in a project.

Could you elaborate on what you think it does better than say Terraform. I looked at their comparison page https://www.pulumi.com/docs/intro/vs/terraform/ and while there is an argument to be made re: being able to write in a more familiar language, its kind of a moot point when config is mostly key value pairs. The biggest paint point with something like Terraform is the fact that it can't actually validate cloud pr…

> (for example declaring an Compute instance with a compute type (string) that doesn't actually exist). Does Pulumi try to address this in some way

Yes, the AWS Pulumi SDK package includes a discriminated union of valid instance types, and the type of the property `aws.ec2.Instance::instanceType` is that union.

(Disclaimer: I am a former maintainer of Terraform and a contributor to Pulumi).

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#150

Post by pantulis »

txcwpalpha wrote: post #53
pantulis wrote: post #46

Earlier quoted context omitted.

Your point is good, but one could suppose Twilio is not a "lone developer".

Certainly not, and in their layers of defense against this stuff, they should have had processes to train people, audit bucket policies, conduct penetration tests etc, and there's no excuse for them not having caught it before. But with that said, one of those layers of defense is also using tools that are easy to keep secure so that the vulnerability doesn't appear in the first place. While Twilio isn't "a lone deve…

I kind of miss the times when working with the cloud was easier or more straightforward.

Now, as has been stated before, reading any cloud vendor documentation is like reading the Oracle manuals of years ago, not a happy experience.

Granted, these clouds now do things much more powerful and target more difficult enterprise scenarios. One can miss Heroku, and AWS has Elastic Beanstalk for that, but EB gets much more difficult fast.

Post reply on HN