Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

theregister.com

Post reply on HN
Print view View original

71–80 of 162 posts

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#71

Post by lozenge »

kirillzubovsky wrote: post #65
whoisjuan wrote: post #28

Having an S3 Bucket with writing permissions to anyone is a major fuck-up. I know many people open buckets to the world at the read-level for whatever reason. I do that myself for static websites. But also writing permissions? This requires a Bucket Policy that is written explicitly to allow this. I don't want to make assumptions because of course we don't know all the information and I personally know very little ab…

Potentially because it was spun 10 years ago while permissions were basic/different/even harder to track, and has since fallen off the wagon? It seems this incident has reminded Twilio to audit all their access right, which is a net positive. Actually, imagine you were an employee who wanted to get this done, but couldn't get management approval? This is one way to fix things ... :)

I remember there are AWS settings to set at account level that override permissive bucket level policies

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#72

Post by mjayhn »

somehnguy wrote: post #57
mjayhn wrote: post #51

Earlier quoted context omitted.

You're using S3 for the first time. Twilio has the capability to hire Ops staff that have used S3 for 10+ years. Throw a penny down the street in any major city and you'll hit a DevOps person with AWS experience. It's amazing to me how often I hear a company not set up s3 properly, or not enable budget alerts on AWS, etc. This is all day 1 AWS stuff. If that means taking away the ability for your developers to even c…

I understand what you're saying but you do realize that a huge number of software shops don't have massive teams or even dedicated OPs people right? The number of heads on my project is ...4. We're it. There is no place or person to just push the work off to. There is no PR approval chain. We regularly meet to sync and discuss plans and thats it. Until very recently all of our infrastructure was in house, right down…

Oh trust me I know totally what you mean, I'm usually the first ops person hired at a startup once they've gotten the budget for one. My goal when I work with devs is to get as much of the infrastructure frustration out of their way as possible (unless they want to help with it) and automate the processes that slow down their code from getting deployed. This usually comes with months of fixing cruft/security issues, etc that could've lead to a "hack" like this when just the devs were running the ship.

I wish MORE small dev teams brought in ops people and didn't wait until they had a staff of 20-30 or whatever it typically winds up being. But I know that's a pipe dream and you're basically losing a developer salary to hire a devops/ops person which isn't the direction they want to go.

edit: typos

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#73

Post by lozenge »

jszymborski wrote: post #63
cj wrote: post #60

Quick shout out to GuardScript which exists to help catch malicious 3rd party (or 1st party) javascript changes: https://www.guardscript.com/ It's a really simple service that fetches a list of script URLs that you provide, and notifies you with the before/after diff whenever the file changes. I found GuardScript through a Show HN a while ago ( https://news.ycombinator.com/item?id=20265141 ) and have been very please…

Isn't this kinda solved by Subresource Integrity (SRI)? https://developer.mozilla.org/en-US/docs/Web/Security/Subres...

Some sites use ad networks that prevent them from setting good SRI policies.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#74

Post by oefrha »

tyingq wrote: post #50

Doesn't sound so non-malicious to me: "Specifically, the modification added code to the end of the TaskRouter.js v1.20 SDK that made an HTTP GET request to hxxps://gold.platinumus.top/track/awswrite?q=dmn and followed the URL returned in the HTML by that request."

> followed the URL returned in the HTML by that request

Translated: we can't possibly know it's non-malicious.

TFA even has this to say:

> And judging from the URL involved, it appears to be an attempt to install a payment-card skimmer – RiskIQ has spotted the same URL in other S3 buckets targeted by miscreants.

Details in the linked blog post[0].

So, very much malicious.

Why the hell they included "non-malicious" in the title (granted, in quotes), I don't know. Readers could have easily dismissed this as actually non-malicious.

[0] https://www.riskiq.com/blog/labs/misconfigured-s3-buckets/

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#75

Post by jaywalk »

lozenge wrote: post #73
jszymborski wrote: post #63

Earlier quoted context omitted.

Isn't this kinda solved by Subresource Integrity (SRI)? https://developer.mozilla.org/en-US/docs/Web/Security/Subres...

Some sites use ad networks that prevent them from setting good SRI policies.

So you don't set the SRI on those scripts. Nothing about the Twilio script referenced by this article would prevent SRI from being applied.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#76

Post by MrBuddyCasino »

kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

I am still amazed some people want to do devops. Write yaml, run script, wait 15 minutes, oh no a typo lets retry. The developer experience is worse than nearly everything that came before it, and I would include ancient PHP and callback-hell nodejs in that. The AWS web UI isn’t much better.

Looking forward to finally try Pulumi in a project.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#77

Post by cj »

jszymborski wrote: post #63
cj wrote: post #60

Quick shout out to GuardScript which exists to help catch malicious 3rd party (or 1st party) javascript changes: https://www.guardscript.com/ It's a really simple service that fetches a list of script URLs that you provide, and notifies you with the before/after diff whenever the file changes. I found GuardScript through a Show HN a while ago ( https://news.ycombinator.com/item?id=20265141 ) and have been very please…

Isn't this kinda solved by Subresource Integrity (SRI)? https://developer.mozilla.org/en-US/docs/Web/Security/Subres...

Not really.

For example, here is Stripe's script tag:

" rel="nofollow">https://js.stripe.com/v3/">

SRI only works for static files whose contents never change. In this example, it would mean that Stripe could never push minor bug fixes to the SDK.

The workaround would be to use static versioned files instead of a generic script URL (or self-host the script), so that the value of the SRI checksum never changes.

Edit: In the case of Twilio, their docs say "By linking directly to our CDN in production, any patch-level (non-breaking) changes made to the SDK will automatically be applied [...]" meaning that if you link to Twilio's CDN, you can't expect the file to be static, and therefore can't use SRI.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#78

Post by kyboren »

corytheboyd wrote: post #35
whoisjuan wrote: post #28

Having an S3 Bucket with writing permissions to anyone is a major fuck-up. I know many people open buckets to the world at the read-level for whatever reason. I do that myself for static websites. But also writing permissions? This requires a Bucket Policy that is written explicitly to allow this. I don't want to make assumptions because of course we don't know all the information and I personally know very little ab…

Yeah this is absolutely inexcusable for a public tech giant. I wonder how incompetently secured the rest of their backend is. This is terrifying, the amount of PII flowing through Twilio is insane, there really needs to be a way to bite companies that completely ignore all of these security compliance checks they say they pass...

To add another data point, consider this email Twilio sent, announcing that they are now forcing all users to use 2FA by October 12th:

> [...] a 2FA verification code sent to your phone (via a call, SMS message, or an authentication app like Authy).

PSTN-based 2FA in 2020 is not a vote of confidence in the security team's competence.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#80

Post by [deleted] »

tyingq wrote: post #50

Doesn't sound so non-malicious to me: "Specifically, the modification added code to the end of the TaskRouter.js v1.20 SDK that made an HTTP GET request to hxxps://gold.platinumus.top/track/awswrite?q=dmn and followed the URL returned in the HTML by that request."

[deleted]
Post reply on HN