Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

theregister.com

Post reply on HN
Print view View original

51–60 of 162 posts

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#51

Post by mjayhn »

somehnguy wrote: post #36
ed25519FUUU wrote: post #8

Earlier quoted context omitted.

Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account. If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL. Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a sin…

It is absolutely a pain. I'm using S3 for the first time on a project right now and literally every time I have to do anything in AWS I end up confused and scared that I'm leaving a wide open security vulnerability. The documentation is complete insanity to anyone just trying to accomplish what should be an extremely common use case in a reasonable amount of time. Before using AWS I was in the 'what morons!' camp whe…

You're using S3 for the first time. Twilio has the capability to hire Ops staff that have used S3 for 10+ years. Throw a penny down the street in any major city and you'll hit a DevOps person with AWS experience. It's amazing to me how often I hear a company not set up s3 properly, or not enable budget alerts on AWS, etc. This is all day 1 AWS stuff. If that means taking away the ability for your developers to even create buckets then do so. A lot of you should just focus on development work and stay far away from operations/sec tasks, and that's fine. I'm going to assume most of these buckets aren't even made via code and are manual. That's bad enough.

I understand someone brand new to S3 being in the dark but someone in your PRs approval chain should have enough Ops skill to know to check these things if you're not going to hire an actual Ops person. And no, none of this is hard for anyone who has ever touched S3. Ops teams have known about these issues for over a decade. It's practically a meme at this point.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#52

Post by mcraiha »

rafaelturk wrote: post #7

S3 security is a pain. S3 options are tricky to master, docs are beyond confusing. A common S3 use case is: Use a Bucket for Read-only static content, (js, html, imgs). From a dev perspective ideally it will work like a protected folder + Web Server, whereas the webserver will have read-only privileges, but reality is far more complicated. I can understand why was easy for Twillio to have this infosec issue, since we…

Is there a service that one could use to check if their bucket accepts public uploads or public ACL changes?

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#53

Post by txcwpalpha »

pantulis wrote: post #46
txcwpalpha wrote: post #44

Earlier quoted context omitted.

The problem with S3 is the exact same problem with everything else in AWS. In their quest to be compatible and attractive to every single possible niche use case of the world's largest companies, they completely forgot the lone developer who just wants to upload a couple files. You'll even see this in their sales pitches. AWS will spend so much time talking about the most advanced use cases and how they are possible,…

Your point is good, but one could suppose Twilio is not a "lone developer".

Certainly not, and in their layers of defense against this stuff, they should have had processes to train people, audit bucket policies, conduct penetration tests etc, and there's no excuse for them not having caught it before.

But with that said, one of those layers of defense is also using tools that are easy to keep secure so that the vulnerability doesn't appear in the first place. While Twilio isn't "a lone developer", I wouldn't be surprised if the person who did create that bucket on behalf of Twilio was just a lone developer fumbling around in the console. And again, that's no excuse, but it still is an area for improvement.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#54

Post by wjossey »

vangelis wrote: post #42
SV_BubbleTime wrote: post #37

Earlier quoted context omitted.

> Never make manual permission changes in the console. Everything is done via SCM’d code. The first time I heard “infrastructure as code” I thought it was some dumb new buzz words, but no, it’s good advice. Your cloud formation, serverless, terraform whatever should be stored in GIT a stack replace is how changes should be made. I consider the console as read-only.

I occasionally spin up things in the AWS console so I can describe them through the CLI and then delete them. IaC is great until you have no idea what the code should be.

Certainly not discouraging this, so thank you for pointing it out.

I do the same, and sometimes I need to do something manually so that I can grok what configurations I need to define in terraform / whatever other tool I'm using. POC'ing is different than a production or even staging service.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#55

Post by paulie_a »

Non malicious? Does the person. That wrote that bit of bullshit even believe it.

They might as well have said "we are not smart enough to figure out the nature of this code. In unrelated news we are hiring senior JavaScript engineers, experience in opsec is a bonus"

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#56

Post by SV_BubbleTime »

nitrogen wrote: post #39
SV_BubbleTime wrote: post #37

Earlier quoted context omitted.

> Never make manual permission changes in the console. Everything is done via SCM’d code. The first time I heard “infrastructure as code” I thought it was some dumb new buzz words, but no, it’s good advice. Your cloud formation, serverless, terraform whatever should be stored in GIT a stack replace is how changes should be made. I consider the console as read-only.

IMO this is code/console dichotomy is a design flaw in most cloud offerings that will haunt the industry for a long time. The web console should be just another UI for editing the configuration in source control. As an example of why that's important, a UI can give you a warning and use affordances to guide beginners away from dangerous setups. In the case of S3, a slider for basic access level can use colors and sho…

> The web console should be just another UI for editing the configuration in source control.

Should. But isn’t.

So until it is, IaC for production. Console or CLI for development.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#57

Post by somehnguy »

mjayhn wrote: post #51
somehnguy wrote: post #36

Earlier quoted context omitted.

It is absolutely a pain. I'm using S3 for the first time on a project right now and literally every time I have to do anything in AWS I end up confused and scared that I'm leaving a wide open security vulnerability. The documentation is complete insanity to anyone just trying to accomplish what should be an extremely common use case in a reasonable amount of time. Before using AWS I was in the 'what morons!' camp whe…

You're using S3 for the first time. Twilio has the capability to hire Ops staff that have used S3 for 10+ years. Throw a penny down the street in any major city and you'll hit a DevOps person with AWS experience. It's amazing to me how often I hear a company not set up s3 properly, or not enable budget alerts on AWS, etc. This is all day 1 AWS stuff. If that means taking away the ability for your developers to even c…

I understand what you're saying but you do realize that a huge number of software shops don't have massive teams or even dedicated OPs people right?

The number of heads on my project is ...4. We're it. There is no place or person to just push the work off to. There is no PR approval chain. We regularly meet to sync and discuss plans and thats it.

Until very recently all of our infrastructure was in house, right down the hall from me. I (or whoever was in that day) took care of making sure it was running smoothly.

Cloud is new to us, and there is no budget to hire a guy who manages AWS full time. That is the case at an insanely high percentage of software shops. Heck I would wager most people outside of 'new tech (sorry, I can't think of a better term)' shops are using point & click in the web UI to manage AWS. There is no code in source control that manages our infrastructure. I'm jealous of the people that get to do it all the 'right way' but for many there isn't enough manpower or time in the day to do it.

Now why we're in this position in the first place is another debate about why management thinks we need to go cloud at all. But its reality for many.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#58

Post by ed25519FUUU »

throwlogon wrote: post #18
ed25519FUUU wrote: post #8

Earlier quoted context omitted.

Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account. If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL. Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a sin…

What you described sounds like a pain.

What’s the pain? You literally don’t do anything to the bucket policy. It’s intentionally blank.

You already have a role in AWS that you use. Go into the console and say “this role can do S3 to this bucket” and you’re done.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#59

Post by natpalmer1776 »

I was curious about the URL that they indicated was non-malicious and so looked it up on google. I found this any.run report:

Report: https://any.run/report/e8c581d52ca3527b61c7ddd9aba5dfeaf9440...

Original URL: h t t p s : / / gold.platinumus.top/track/awswrite?q=dmn

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#60

Post by cj »

Quick shout out to GuardScript which exists to help catch malicious 3rd party (or 1st party) javascript changes: https://www.guardscript.com/

It's a really simple service that fetches a list of script URLs that you provide, and notifies you with the before/after diff whenever the file changes.

I found GuardScript through a Show HN a while ago (https://news.ycombinator.com/item?id=20265141) and have been very pleased with it.

Post reply on HN