Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

theregister.com

Post reply on HN
Print view View original

121–130 of 162 posts

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#121

Post by MrBuddyCasino »

marcocampos wrote: post #81
MrBuddyCasino wrote: post #76

Earlier quoted context omitted.

I am still amazed some people want to do devops. Write yaml, run script, wait 15 minutes, oh no a typo lets retry. The developer experience is worse than nearly everything that came before it, and I would include ancient PHP and callback-hell nodejs in that. The AWS web UI isn’t much better. Looking forward to finally try Pulumi in a project.

How will Pulumi fix the issues you mentioned?

By doing what XML-with-schema did 15 years ago: give compile-time type checking, making sure at least the syntax and types are valid.

Ever had CloudFormation yell at you because some function like "Sub!" only accepts alphanumeric characters? Created complicated constructs because there are no variables, and very limited conditionals? Copy-pasted templates like an insane person because there is no way to abstract anything?

Not my idea of fun. All of this is solved by using a proper programming language.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#122

Post by aldarisbm »

somehnguy wrote: post #57
mjayhn wrote: post #51

Earlier quoted context omitted.

You're using S3 for the first time. Twilio has the capability to hire Ops staff that have used S3 for 10+ years. Throw a penny down the street in any major city and you'll hit a DevOps person with AWS experience. It's amazing to me how often I hear a company not set up s3 properly, or not enable budget alerts on AWS, etc. This is all day 1 AWS stuff. If that means taking away the ability for your developers to even c…

I understand what you're saying but you do realize that a huge number of software shops don't have massive teams or even dedicated OPs people right? The number of heads on my project is ...4. We're it. There is no place or person to just push the work off to. There is no PR approval chain. We regularly meet to sync and discuss plans and thats it. Until very recently all of our infrastructure was in house, right down…

I totally understand where this is coming from, but even if we factor AWS out. IAM _IS_ difficult to implement, there really isn't a way to make it "simple" while still make it extremely granular. You can't have without the other.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#123

Post by ericbarrett »

bjclark wrote: post #87
ericbarrett wrote: post #85

Earlier quoted context omitted.

I really have to disagree here, at least with your conclusions re complexity. No doubt AWS S3 configuration is very technical and has a lot of docs and corner cases. However, if you are a company like Twilio, and S3 is front-and-center the source of your business (and if it's serving your SDK, it probably is), your S3 bucket permissions are clearly worthy of periodic review. And you should have this done by somebody…

The flaw in your logic is that "S3 is front-and-center the source of your business" is never true with AWS. To do anything in AWS there's at least 3 or 4 vaguely connected services, including IAM and RAM as a completely separate UIs. Usually with S3 you also have Cloudfront or some other CDN. You probably also have Cloudwatch logs and CLoudtrail event tracking. You might have VPCs involved. Some or all of these thing…

These only seem “vaguely connected” if you haven’t studied them together. IAM and S3 are intricately intertwined and you should understand both at at least an intermediate level to work with them in sensitive environments. (Note that IAM has permissions specific to each Amazon service, all documented in detail.) None of the rest of the systems you mentioned gate write access, though certainly if they’re in your production flow you should study and review them as well.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#124

Post by cwhiz »

kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

I agree to an extent. AWS is beginning to feel like a duct taped spaghetti mess. It is becoming increasingly difficult to do things “right” without getting deep into their confusing documentation.

However, I’m not necessarily sure that it is the wrong approach. You can use AWS at almost any level you like and if you go deep into the woods with any service you’re eventually going to find yourself buried in documentation.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#125

Post by bjclark »

ericbarrett wrote: post #123
bjclark wrote: post #87

Earlier quoted context omitted.

The flaw in your logic is that "S3 is front-and-center the source of your business" is never true with AWS. To do anything in AWS there's at least 3 or 4 vaguely connected services, including IAM and RAM as a completely separate UIs. Usually with S3 you also have Cloudfront or some other CDN. You probably also have Cloudwatch logs and CLoudtrail event tracking. You might have VPCs involved. Some or all of these thing…

These only seem “vaguely connected” if you haven’t studied them together. IAM and S3 are intricately intertwined and you should understand both at at least an intermediate level to work with them in sensitive environments. (Note that IAM has permissions specific to each Amazon service, all documented in detail.) None of the rest of the systems you mentioned gate write access, though certainly if they’re in your produ…

> all documented in detail

Thanks for the giggle.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#126

Post by [deleted] »

aldarisbm wrote: post #119
morvita wrote: post #114

Earlier quoted context omitted.

> and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform. I love Terraform, but the fact that you require an entirely different company's orchestration tools to make AWS "not that hard to orchestrate" is evidence of how poor the UX of AWS is.

But terraform is using the aws sdk behind the covers. Most of the stuff available with terraform is available with CloudFormation minus a thing or two. (takes them a minute to catch-up to other teams new services/features)

[deleted]

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#127

Post by boarnoah »

MrBuddyCasino wrote: post #76
kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

I am still amazed some people want to do devops. Write yaml, run script, wait 15 minutes, oh no a typo lets retry. The developer experience is worse than nearly everything that came before it, and I would include ancient PHP and callback-hell nodejs in that. The AWS web UI isn’t much better. Looking forward to finally try Pulumi in a project.

Could you elaborate on what you think it does better than say Terraform. I looked at their comparison page https://www.pulumi.com/docs/intro/vs/terraform/ and while there is an argument to be made re: being able to write in a more familiar language, its kind of a moot point when config is mostly key value pairs.

The biggest paint point with something like Terraform is the fact that it can't actually validate cloud provider specific problems (for example declaring an Compute instance with a compute type (string) that doesn't actually exist). Does Pulumi try to address this in some way (granted its much easier to write your own checks like that in language as opposed to writing a go module for Terraform providers).

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#128

Post by orf »

morvita wrote: post #114
orf wrote: post #110

Earlier quoted context omitted.

Each of these services are highly different concerns and don’t belong together, and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform. If you’ve got one team managing your logging who don’t talk to whatever team who manages your s3/cloudfront setup, then that’s your problem.

> and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform. I love Terraform, but the fact that you require an entirely different company's orchestration tools to make AWS "not that hard to orchestrate" is evidence of how poor the UX of AWS is.

You can use Cloudformation. Like, what do you want here? Some super console that nobody asked for because anyone who knows what they are doing knows why you want infrastructure as code?

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#129

Post by disiplus »

kyboren wrote: post #78
corytheboyd wrote: post #35

Earlier quoted context omitted.

Yeah this is absolutely inexcusable for a public tech giant. I wonder how incompetently secured the rest of their backend is. This is terrifying, the amount of PII flowing through Twilio is insane, there really needs to be a way to bite companies that completely ignore all of these security compliance checks they say they pass...

To add another data point, consider this email Twilio sent, announcing that they are now forcing all users to use 2FA by October 12th: > [...] a 2FA verification code sent to your phone (via a call, SMS message, or an authentication app like Authy). PSTN-based 2FA in 2020 is not a vote of confidence in the security team's competence.

even stripe offers SMS for 2FA. both PSTN and SMS are not perfect and have similar downsides.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#130

Post by disiplus »

u801e wrote: post #92
kyboren wrote: post #78

Earlier quoted context omitted.

To add another data point, consider this email Twilio sent, announcing that they are now forcing all users to use 2FA by October 12th: > [...] a 2FA verification code sent to your phone (via a call, SMS message, or an authentication app like Authy). PSTN-based 2FA in 2020 is not a vote of confidence in the security team's competence.

> PSTN-based 2FA in 2020 is not a vote of confidence in the security team's competence. A lot of companies go with email and/or PSTN based 2FA for customer convenience and to get them to use 2FA for their accounts. I try to avoid using 2FA for accounts that only offer PSTN or email based 2FA because it's less secure than my randomly generated long password. It would be nice if more services offered 2FA based on clien…

i would not say it's less secure. Even if somebody overtakes one of my machine and gets to my lastpass vault or steals my clipboard having my smartphone with me and a SMS or PSTN based 2FA is better then not having it at all.
Post reply on HN