Live data from Hacker News

Twitter has an internal root CA problem

izzodlaw.com

71–79 of 79 posts

Re: Twitter has an internal root CA problem

#71
post #64

I think people who work in reliability see this type of thing as the real existential threat to twitter. It's unrealistic that a large infrastructure would fall over overnight, but what is very realistic is small problems being neglected until they become big problems, or multiple problems happening at the same time. This alone is probably manageable, it might even be simple but painful to handle for 2-15 of twitters…

That's okay, Musk tweeted that Twitter needs a complete, green-field rewrite 5 days ago, I'm sure that will solve the problem.

How many engineers would that take Michael? 4, hardcore, over the weekend?

Re: Twitter has an internal root CA problem

#72

Earlier quoted context omitted.

when building secure systems one of the key principles is assume someone will leak the private key. this is how we get to hsm maybe another one is assume you will lose access to the hsm. sure spinning up a new trust chain is annoying but it wouldn’t take that long to do. totally agree this post is overblown

spinning up a new trust chain is not so hard, but deploying that trust chain to thousands of servers around the world when your automation tool isn't available to do it with is really, really hard.

This is why I've been very skeptical of the kids these days kicking literally everyone off of the production servers.

Having a few greybeards with the keys to the kingdom and the wisdom not to use it to screw around in prod, outside of existential emergencies, can be quite useful.

Also should have console access.

One time a bad config push took out a couple hundred webservers with effectively a single iptables default deny rule and we had to get a dozen people to fix them in chunks by logging in manually over remote terminal (probably could have expect-scripted that up, but it was quicker to just get it done).

Re: Twitter has an internal root CA problem

#73
post #55

Earlier quoted context omitted.

So far, there seem to be surprisingly few issues. Some glitches here and there, but overall stability looks still quite good. I would've expected major issues much sooner, especially as they did push out new features in the meantime.

Do you use it actively? There have been minor problems on a near daily basis and moderate ones a handful of times. It was only ever average stable before but it was at least always that, it's far less consistent recently.

Do you have data on this? Its not like other apps don't have issues. Sometimes I open netflix and it takes 30 seconds to show my profiles; that doesn't mean the app is garbage.

Re: Twitter has an internal root CA problem

#74

Earlier quoted context omitted.

Do you use it actively? There have been minor problems on a near daily basis and moderate ones a handful of times. It was only ever average stable before but it was at least always that, it's far less consistent recently.

Do you have data on this? Its not like other apps don't have issues. Sometimes I open netflix and it takes 30 seconds to show my profiles; that doesn't mean the app is garbage.

Do I have data on a website I use sometimes? No.

Re: Twitter has an internal root CA problem

#75
It's pretty amazing that a formerly public company like Twitter had such shitty documentation/processes/infrastructure.

I thought SOX mandated this sort of internal controls - after all, Twitter basically seems to be full of infrastructure risks that would (and have) negatively impacted them financially in a material way.

No key access? Why didn't they print it out and stick it in a safe deposit box, which is what a couple of startups I've been with have done...along with a couple of other key pieces of paper. Physical backup.

Re: Twitter has an internal root CA problem

#76
The really interesting part of this is what else is tied to that CA. If it’s just Puppet, it’s bad enough; internal PKIs have a habit of metastasizing into lots of other places, though, precisely because everything internal trusts them. Worst-case here is that some piece of the internals of the Twitter app relies on things from that CA—-for instance, it relies on packages to do app config changes or updates and the packages have to be signed from that chain or served from something with a cert from it. In that case they’d be hosed: you’d have to replace every copy of the Twitter app. Fairly unlikely, but wouldn’t be the first time I’ve seen it happen.

Beyond that, though: Internal build systems? Data encryption? User client auth to critical services? Internal app mTLS for data exchanges? The list of possibilities goes on and on…

Re: Twitter has an internal root CA problem

#77

> Musk fired everyone with access to the private key to their internal root CA, The way forward is to generate a new CA root certificate. > and they can no longer run puppet because the puppet master's CA cert expired They can reconfigure internal tools to use the new CA root certificate, or rather one of the signed intermediate certificates. > and they can't get a new one because no one has access. They can simply g…

>Twitter was so awful before, that it relied on people to safeguard the keys to the kingdom. This is very bad practice...that can be accessed by the board of directors, the executive managers, and designated maintainers. LOL, you realize all the PEOPLE you list as the PEOPLE who should be able to manage the keys to the kingdom are PEOPLE? Board of directors - fired on day one of Musk takeover, executive managers - ma…

According to the latest publicly available sources there are still many hundreds of folks on the active payroll at Twitter, do you know of any evidence to the contrary?

Re: Twitter has an internal root CA problem

#78

> Musk fired everyone with access to the private key to their internal root CA, The way forward is to generate a new CA root certificate. > and they can no longer run puppet because the puppet master's CA cert expired They can reconfigure internal tools to use the new CA root certificate, or rather one of the signed intermediate certificates. > and they can't get a new one because no one has access. They can simply g…

> For any mission critical assets, and especially certificates, but also passwords... current modern day corporate practice is to have a secure ledger of these that can be accessed by the board of directors, the executive managers, and designated maintainers. At no point ever should the password be entrusted to anybody, but rather a "role" that functions as the one who has access. Say for example, the CIO/CTO and their subordinates.

Maybe in hacker movies. In real life, you try your best to avoid anyone having access to keys or passwords, and rely on HSMs, cloud KMS, secret services, etc. Access to those things is controlled by your security team, with multi-factor authentication, often stored in safes, with alerts being fired when they are used (because they should never be used). The audit logs that trigger these alerts should be written in WORM storage, so you can track access back down to individuals, and so that you know when you need to rotate secrets accessed by humans. Ideally your CA infrastructure automatically rotates and distributes.

There's absolutely no way in hell you should allow your board to have access to these things.

Most companies slowly work their way towards full automation, and until that happens, your security team usually owns manual rotations of critical systems like this. Only a fucking moron would fire all of these people.

Re: Twitter has an internal root CA problem

#79
post #55

Earlier quoted context omitted.

So far, there seem to be surprisingly few issues. Some glitches here and there, but overall stability looks still quite good. I would've expected major issues much sooner, especially as they did push out new features in the meantime.

Do you use it actively? There have been minor problems on a near daily basis and moderate ones a handful of times. It was only ever average stable before but it was at least always that, it's far less consistent recently.

I use it actively and feel like recommendations have improved, at least for me. Ads are a bit more annoying but I've faced zero problems with stability. But I'm only reading, not actively tweeting.
Post reply on HN