I think people who work in reliability see this type of thing as the real existential threat to twitter. It's unrealistic that a large infrastructure would fall over overnight, but what is very realistic is small problems being neglected until they become big problems, or multiple problems happening at the same time. This alone is probably manageable, it might even be simple but painful to handle for 2-15 of twitters…
That's okay, Musk tweeted that Twitter needs a complete, green-field rewrite 5 days ago, I'm sure that will solve the problem.
Twitter has an internal root CA problem
71–79 of 79 posts
Re: Twitter has an internal root CA problem
#72Earlier quoted context omitted.
when building secure systems one of the key principles is assume someone will leak the private key. this is how we get to hsm maybe another one is assume you will lose access to the hsm. sure spinning up a new trust chain is annoying but it wouldn’t take that long to do. totally agree this post is overblown
spinning up a new trust chain is not so hard, but deploying that trust chain to thousands of servers around the world when your automation tool isn't available to do it with is really, really hard.
Having a few greybeards with the keys to the kingdom and the wisdom not to use it to screw around in prod, outside of existential emergencies, can be quite useful.
Also should have console access.
One time a bad config push took out a couple hundred webservers with effectively a single iptables default deny rule and we had to get a dozen people to fix them in chunks by logging in manually over remote terminal (probably could have expect-scripted that up, but it was quicker to just get it done).
Re: Twitter has an internal root CA problem
#73Earlier quoted context omitted.
So far, there seem to be surprisingly few issues. Some glitches here and there, but overall stability looks still quite good. I would've expected major issues much sooner, especially as they did push out new features in the meantime.
Do you use it actively? There have been minor problems on a near daily basis and moderate ones a handful of times. It was only ever average stable before but it was at least always that, it's far less consistent recently.
Re: Twitter has an internal root CA problem
#74Earlier quoted context omitted.
Do you use it actively? There have been minor problems on a near daily basis and moderate ones a handful of times. It was only ever average stable before but it was at least always that, it's far less consistent recently.
Do you have data on this? Its not like other apps don't have issues. Sometimes I open netflix and it takes 30 seconds to show my profiles; that doesn't mean the app is garbage.
Re: Twitter has an internal root CA problem
#75I thought SOX mandated this sort of internal controls - after all, Twitter basically seems to be full of infrastructure risks that would (and have) negatively impacted them financially in a material way.
No key access? Why didn't they print it out and stick it in a safe deposit box, which is what a couple of startups I've been with have done...along with a couple of other key pieces of paper. Physical backup.
Re: Twitter has an internal root CA problem
#76Beyond that, though: Internal build systems? Data encryption? User client auth to critical services? Internal app mTLS for data exchanges? The list of possibilities goes on and on…
Re: Twitter has an internal root CA problem
#77> Musk fired everyone with access to the private key to their internal root CA, The way forward is to generate a new CA root certificate. > and they can no longer run puppet because the puppet master's CA cert expired They can reconfigure internal tools to use the new CA root certificate, or rather one of the signed intermediate certificates. > and they can't get a new one because no one has access. They can simply g…
>Twitter was so awful before, that it relied on people to safeguard the keys to the kingdom. This is very bad practice...that can be accessed by the board of directors, the executive managers, and designated maintainers. LOL, you realize all the PEOPLE you list as the PEOPLE who should be able to manage the keys to the kingdom are PEOPLE? Board of directors - fired on day one of Musk takeover, executive managers - ma…
Re: Twitter has an internal root CA problem
#78> Musk fired everyone with access to the private key to their internal root CA, The way forward is to generate a new CA root certificate. > and they can no longer run puppet because the puppet master's CA cert expired They can reconfigure internal tools to use the new CA root certificate, or rather one of the signed intermediate certificates. > and they can't get a new one because no one has access. They can simply g…
Maybe in hacker movies. In real life, you try your best to avoid anyone having access to keys or passwords, and rely on HSMs, cloud KMS, secret services, etc. Access to those things is controlled by your security team, with multi-factor authentication, often stored in safes, with alerts being fired when they are used (because they should never be used). The audit logs that trigger these alerts should be written in WORM storage, so you can track access back down to individuals, and so that you know when you need to rotate secrets accessed by humans. Ideally your CA infrastructure automatically rotates and distributes.
There's absolutely no way in hell you should allow your board to have access to these things.
Most companies slowly work their way towards full automation, and until that happens, your security team usually owns manual rotations of critical systems like this. Only a fucking moron would fire all of these people.
Re: Twitter has an internal root CA problem
#79Earlier quoted context omitted.
So far, there seem to be surprisingly few issues. Some glitches here and there, but overall stability looks still quite good. I would've expected major issues much sooner, especially as they did push out new features in the meantime.
Do you use it actively? There have been minor problems on a near daily basis and moderate ones a handful of times. It was only ever average stable before but it was at least always that, it's far less consistent recently.