Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

theregister.com

Post reply on HN
Print view View original

61–70 of 162 posts

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#61

Post by kirillzubovsky »

I love AWS as any other developer, but this is their fault.

They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs.

Amazon's logic is probably to provide the piping, and leave the rest of the end user, but at this point it's not enough. In my opinion, they need to heavily invest into the customer-facing layer of their applications that shows all the status, gaps and holes, or these things keep happening.

Then again, they can continue as is, and the incidents will keep as is, and as long as nothing implodes and no big customer leaves, financially it makes sense to do nothing. Just thinking out loud.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#62

Post by SteveNuts »

somehnguy wrote: post #57
mjayhn wrote: post #51

Earlier quoted context omitted.

You're using S3 for the first time. Twilio has the capability to hire Ops staff that have used S3 for 10+ years. Throw a penny down the street in any major city and you'll hit a DevOps person with AWS experience. It's amazing to me how often I hear a company not set up s3 properly, or not enable budget alerts on AWS, etc. This is all day 1 AWS stuff. If that means taking away the ability for your developers to even c…

I understand what you're saying but you do realize that a huge number of software shops don't have massive teams or even dedicated OPs people right? The number of heads on my project is ...4. We're it. There is no place or person to just push the work off to. There is no PR approval chain. We regularly meet to sync and discuss plans and thats it. Until very recently all of our infrastructure was in house, right down…

To me this is like buying a high end sports car, and neglecting the maintenance because you never factored in the expensive maintenance, and therefore the true cost of what you're buying.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#63

Post by jszymborski »

cj wrote: post #60

Quick shout out to GuardScript which exists to help catch malicious 3rd party (or 1st party) javascript changes: https://www.guardscript.com/ It's a really simple service that fetches a list of script URLs that you provide, and notifies you with the before/after diff whenever the file changes. I found GuardScript through a Show HN a while ago ( https://news.ycombinator.com/item?id=20265141 ) and have been very please…

Isn't this kinda solved by Subresource Integrity (SRI)?

https://developer.mozilla.org/en-US/docs/Web/Security/Subres...

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#64

Post by kirillzubovsky »

cj wrote: post #60

Quick shout out to GuardScript which exists to help catch malicious 3rd party (or 1st party) javascript changes: https://www.guardscript.com/ It's a really simple service that fetches a list of script URLs that you provide, and notifies you with the before/after diff whenever the file changes. I found GuardScript through a Show HN a while ago ( https://news.ycombinator.com/item?id=20265141 ) and have been very please…

What a great example of a company that figured out a way to put a utility layer on top of existing tools and monetize it. Thanks for sharing.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#65

Post by kirillzubovsky »

whoisjuan wrote: post #28

Having an S3 Bucket with writing permissions to anyone is a major fuck-up. I know many people open buckets to the world at the read-level for whatever reason. I do that myself for static websites. But also writing permissions? This requires a Bucket Policy that is written explicitly to allow this. I don't want to make assumptions because of course we don't know all the information and I personally know very little ab…

Potentially because it was spun 10 years ago while permissions were basic/different/even harder to track, and has since fallen off the wagon?

It seems this incident has reminded Twilio to audit all their access right, which is a net positive.

Actually, imagine you were an employee who wanted to get this done, but couldn't get management approval? This is one way to fix things ... :)

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#66

Post by somehnguy »

SteveNuts wrote: post #62
somehnguy wrote: post #57

Earlier quoted context omitted.

I understand what you're saying but you do realize that a huge number of software shops don't have massive teams or even dedicated OPs people right? The number of heads on my project is ...4. We're it. There is no place or person to just push the work off to. There is no PR approval chain. We regularly meet to sync and discuss plans and thats it. Until very recently all of our infrastructure was in house, right down…

To me this is like buying a high end sports car, and neglecting the maintenance because you never factored in the expensive maintenance, and therefore the true cost of what you're buying.

I agree. People do it all the time though.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#67

Post by jfoutz »

ljoshua wrote: post #38
TheSpiciestDev wrote: post #34

Earlier quoted context omitted.

"In local news, stranger off the street walks through residence's open door and moves around furniture."

Well, even in such a world, that would still be considered trespassing.

I am not a lawyer, I'm definitely not your lawyer. Trespassing and possibly theft, I think. Theft includes moving stuff without the owners permission. so if I ,say, have a tow truck and move a car across the street without the owners consent, I'm a thief.

I only know this because a friend talked about a case where they rotated a car in-place, and there was a question about is that theft? its center of mass is still where they left it, so it wound up not being theft. Moving furniture in a house is a weird one. Probably leans on ill intent, moving a chair to block a door seems like it's pointed to giving the owner a hard time. moving a chair to perform cpr has much purer motives.

So anyway, there's my ill-informed view.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#68

Post by fisherjeff »

somehnguy wrote: post #36
ed25519FUUU wrote: post #8

Earlier quoted context omitted.

Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account. If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL. Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a sin…

It is absolutely a pain. I'm using S3 for the first time on a project right now and literally every time I have to do anything in AWS I end up confused and scared that I'm leaving a wide open security vulnerability. The documentation is complete insanity to anyone just trying to accomplish what should be an extremely common use case in a reasonable amount of time. Before using AWS I was in the 'what morons!' camp whe…

It's gotten better, if you can believe it. Regardless, I've managed to come up with one foolproof method for securing my data: Never put anything important on S3.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#69

Post by mwarkentin »

mcraiha wrote: post #52
rafaelturk wrote: post #7

S3 security is a pain. S3 options are tricky to master, docs are beyond confusing. A common S3 use case is: Use a Bucket for Read-only static content, (js, html, imgs). From a dev perspective ideally it will work like a protected folder + Web Server, whereas the webserver will have read-only privileges, but reality is far more complicated. I can understand why was easy for Twillio to have this infosec issue, since we…

Is there a service that one could use to check if their bucket accepts public uploads or public ACL changes?

Aws S3 surfaces this info now, as well as allowing you to enforce that they’re disabled at the bucket or account level (public access blocks).

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#70

Post by sgt-serverless »

txcwpalpha wrote: post #44
somehnguy wrote: post #36

Earlier quoted context omitted.

It is absolutely a pain. I'm using S3 for the first time on a project right now and literally every time I have to do anything in AWS I end up confused and scared that I'm leaving a wide open security vulnerability. The documentation is complete insanity to anyone just trying to accomplish what should be an extremely common use case in a reasonable amount of time. Before using AWS I was in the 'what morons!' camp whe…

The problem with S3 is the exact same problem with everything else in AWS. In their quest to be compatible and attractive to every single possible niche use case of the world's largest companies, they completely forgot the lone developer who just wants to upload a couple files. You'll even see this in their sales pitches. AWS will spend so much time talking about the most advanced use cases and how they are possible,…

We have been solving this issue with the app I have built Commandeer. Here is the main S3 dashboard https://imgur.com/gallery/JutGUI3 And here is the view of an individual bucket https://imgur.com/gallery/0OlKY7Q . It let's you completely manage S3 and 25 other AWS and Cloud services from a desktop app. You can easily see permissions and also preview every type of file. We have also done this for DynamoDB, Lambda, CloudWatch Alarms/Logs/Rules. The tool was built out of our frustration that many have.
Post reply on HN