Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

theregister.com

Post reply on HN
Print view View original

91–100 of 162 posts

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#91

Post by txcwpalpha »

ericbarrett wrote: post #85
kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

I really have to disagree here, at least with your conclusions re complexity. No doubt AWS S3 configuration is very technical and has a lot of docs and corner cases. However, if you are a company like Twilio, and S3 is front-and-center the source of your business (and if it's serving your SDK, it probably is), your S3 bucket permissions are clearly worthy of periodic review. And you should have this done by somebody…

>That said, I don't blame Twilio for not catching this

I do.

Twilio is a multi-billion dollar company and there is no excuse for them not having proper security processes to catch stuff like this early. Even if we take the "S3 is hard" arguments at face value, this wasn't a 0-day or some complicated unpredictable exploit. This was an extremely basic misconfiguration on a mission-critical part of their architecture that would have been caught by even the simplest out-of-the-box penetration test or audit. For a company like Twilio to not be doing basic, fundamental stuff like that is a big deal and they certainly should be blamed for it.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#92

Post by u801e »

kyboren wrote: post #78
corytheboyd wrote: post #35

Earlier quoted context omitted.

Yeah this is absolutely inexcusable for a public tech giant. I wonder how incompetently secured the rest of their backend is. This is terrifying, the amount of PII flowing through Twilio is insane, there really needs to be a way to bite companies that completely ignore all of these security compliance checks they say they pass...

To add another data point, consider this email Twilio sent, announcing that they are now forcing all users to use 2FA by October 12th: > [...] a 2FA verification code sent to your phone (via a call, SMS message, or an authentication app like Authy). PSTN-based 2FA in 2020 is not a vote of confidence in the security team's competence.

> PSTN-based 2FA in 2020 is not a vote of confidence in the security team's competence.

A lot of companies go with email and/or PSTN based 2FA for customer convenience and to get them to use 2FA for their accounts.

I try to avoid using 2FA for accounts that only offer PSTN or email based 2FA because it's less secure than my randomly generated long password. It would be nice if more services offered 2FA based on client side TLS certificates.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#94

Post by rumanator »

kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

> They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it.

In AWS you need to explicitly set a bucket as publicly accessible, which requires you to go through a couple of dialogs where you're expected to say "yes, yes I want to allow others to access this S3 bucket."

Afterwards you still get a bucket that does not have write access by default. To allow write access you need to specifically set bucket access policies that explicitly allow others to write to your bucket.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#95

Post by jedberg »

kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

I disagree. They started by just providing the plumbing, and Twillio is using that plumbing.

But today if you make an S3 bucket, they take you through a wizard that makes it secure by default. They also run audits for free and send you the results.

They also have Lightsail that just does everything for you securely.

They have lots of user friendly options.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#96

Post by benmmurphy »

jfoutz wrote: post #67
ljoshua wrote: post #38

Earlier quoted context omitted.

Well, even in such a world, that would still be considered trespassing.

I am not a lawyer, I'm definitely not your lawyer. Trespassing and possibly theft, I think. Theft includes moving stuff without the owners permission. so if I ,say, have a tow truck and move a car across the street without the owners consent, I'm a thief. I only know this because a friend talked about a case where they rotated a car in-place, and there was a question about is that theft? its center of mass is still w…

I thought in common law countries theft required you to have the intent to permanently deprive the victim of the property. this is why there are separate statutes to cover stealing cars because people just claimed they were taking it temporarily.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#97

Post by Someone1234 »

untog wrote: post #26
pquerna wrote: post #15

Earlier quoted context omitted.

Yes! Subresource Integrity is the exact counter-measure to this kind of attack: https://developer.mozilla.org/en-US/docs/Web/Security/Subres... However, on Twilio's documentation site, they do not include the integrity attribute in their examples: https://www.twilio.com/docs/taskrouter/js-sdk/workspace/task...

It's difficult. I like the idea of subresource integrity a lot, but a counter example: Twilio discovers a bug in their JS SDK. Without subresource integrity they can push up a fix and propagate it to all clients immediately. With subresource integrity they would need every client to update their web site to change the integrity attribute. If it was a serious bug then that could be a huge issue. Even putting the secur…

> Without subresource integrity they can push up a fix and propagate it to all clients immediately.

In reality, no you cannot.

You have multiple layers of cache between the S3 bucket and rendering, unless you disable caching entirely at a massive increased cost. Some of these caches are poorly behaving (e.g. intermediary caching).

The correct way of doing this ALREADY is to increment the version number in the URL (e.g. /3.1.0/My.Lib.js to /3.1.1/My.Lib.js) and to re-point the pages to use the updated library. This is reliably cache breaking, and will assure end users are getting the bug fixed version faster (or ever in some cases).

Once you're already doing it correctly, Subresource Integrity is a freebie. There's a reason why "sdk-latest.js" is largely dead concept from a bygone era: it is a huge anti-pattern and anti-feature.

Heck allowing upstream to blindly push you changes without being informed is nuts regardless. This argument is essentially: "I'm doing it wrong, and Subresource Integrity would stop that, so it is a non-starter." Without considering that what you're doing is bad practice before Subresource Integrity joined the party.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#98

Post by untog »

sterkekoffie wrote: post #47
untog wrote: post #26

Earlier quoted context omitted.

It's difficult. I like the idea of subresource integrity a lot, but a counter example: Twilio discovers a bug in their JS SDK. Without subresource integrity they can push up a fix and propagate it to all clients immediately. With subresource integrity they would need every client to update their web site to change the integrity attribute. If it was a serious bug then that could be a huge issue. Even putting the secur…

>If it was a serious bug then that could be a huge issue. Regardless of the severity of the bug, the only-case scenario is that all the sites you have pulling from that CDN break until you recompute the hash. How annoying this is scales directly to how frequently your libraries have to release vital security bugfixes.

> the only-case scenario is that all the sites you have pulling from that CDN break until you recompute the hash

Until you recompute the hash and communicate that new hash to them and they implement it on their site. It’s not nothing from an implementation point of view.

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#99

Post by politelemon »

kirillzubovsky wrote: post #61

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably…

I'm not seeing any UX problems here. They configured an S3 access policy incorrectly while troubleshooting, 5 years ago.

FTFA:

> During our incident review, we identified that this path was not initially configured with public write access when it was added in 2015. We implemented a change 5 months later while troubleshooting a problem with one of our build systems and the permissions on that path were not properly reset once the issue had been fixed.

> { "Sid": "AllowPublicRead", "Effect": "Allow", "Principal": { "AWS": "" }, "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": "arn:aws:s3:::media.twiliocdn.com/taskrouter/" }

Re: Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

#100

Post by dsl »

FDSGSG wrote: post #83
oefrha wrote: post #74

Earlier quoted context omitted.

> followed the URL returned in the HTML by that request Translated: we can't possibly know it's non-malicious. TFA even has this to say: > And judging from the URL involved, it appears to be an attempt to install a payment-card skimmer – RiskIQ has spotted the same URL in other S3 buckets targeted by miscreants. Details in the linked blog post[0]. So, very much malicious. Why the hell they included "non-malicious" in…

FWIW, that's a redirector and not a skimmer.

Magecart is a well known framework in the underground for stealing credit card numbers from websites (if they can get the JavaScript running on a checkout page).

If you hit it directly they will redirect to a random "Spin the wheel win a prize" type site in an attempt to hide the malicious nature of the payload.

Post reply on HN