Earlier quoted context omitted.
We have 7 racks and 3 people working in ops and built Puppet setup "right". It's not hard. And their setup was probably right too. Just that nobody plans for "bus hit our entire ops team"
More commonly you don't plan for it, you make sure the entire ops team are never on the same bus, same plane, preferably not even in the same city.
Twitter has an internal root CA problem
51–60 of 79 posts
Re: Twitter has an internal root CA problem
#52If this is true - who knows - then it reflects rather badly on the people who were fired - as they didn't implement safeguard for a 'run over by a bus' scenario when they were in charge.
Or they had a "run over by a bus" scenario that assumed that the entire team wasn't going to be run over by a bus all at the same time?
Re: Twitter has an internal root CA problem
#53Earlier quoted context omitted.
Create new root CA, ssh to machine, remove old certs, re-add machine to Puppet, sign the new CSR on Puppet master, then it will download new root. Very simple operation... if you have working SSH access with root. If they don't, well...
If they're in the cloud, it's pretty straightforward to re-mount the drive somewhere else and replace the SSH keys.
Re: Twitter has an internal root CA problem
#54If this is true - who knows - then it reflects rather badly on the people who were fired - as they didn't implement safeguard for a 'run over by a bus' scenario when they were in charge.
Re: Twitter has an internal root CA problem
#55[flagged]
Re: Twitter has an internal root CA problem
#56[flagged]
So far, there seem to be surprisingly few issues. Some glitches here and there, but overall stability looks still quite good. I would've expected major issues much sooner, especially as they did push out new features in the meantime.
Re: Twitter has an internal root CA problem
#57The way forward is to generate a new CA root certificate.
> and they can no longer run puppet because the puppet master's CA cert expired
They can reconfigure internal tools to use the new CA root certificate, or rather one of the signed intermediate certificates.
> and they can't get a new one because no one has access.
They can simply generate new CA root certificates, and sign or create new intermediate certificates.
> They no longer can mint certs.
Yes, they, can...
> My limited understanding in this area is that this is...very bad
No, it, is, not...
There are two immediate issues that come to mind.
* Twitter was so awful before, that it relied on people to safeguard the keys to the kingdom. This is very bad practice, and one of the many things Musk will no doubt be fixing. For any mission critical assets, and especially certificates, but also passwords... current modern day corporate practice is to have a secure ledger of these that can be accessed by the board of directors, the executive managers, and designated maintainers. At no point ever should the password be entrusted to anybody, but rather a "role" that functions as the one who has access. Say for example, the CIO/CTO and their subordinates.
* The Second issue is the one everyone is fixating upon, and that's firing important people who put the company at risk. This is a big issue, and certainly Musk could have done a better job of scoping out who represents a single-point-of- failure at twitter, eliminate that risk, and then proceed with the culling. In a modern enterprise no single person should be capable of putting the entire operation at risk. It's just that simple. So in a way, Musk accelerated what was probably inevitable at Twitter already. They were probably precariously close to destruction already, and now they can learn the hard way of not repeating these mistakes.
Re: Twitter has an internal root CA problem
#58> Musk fired everyone with access to the private key to their internal root CA, The way forward is to generate a new CA root certificate. > and they can no longer run puppet because the puppet master's CA cert expired They can reconfigure internal tools to use the new CA root certificate, or rather one of the signed intermediate certificates. > and they can't get a new one because no one has access. They can simply g…
maybe another one is assume you will lose access to the hsm. sure spinning up a new trust chain is annoying but it wouldn’t take that long to do. totally agree this post is overblown
Re: Twitter has an internal root CA problem
#59> Musk fired everyone with access to the private key to their internal root CA, The way forward is to generate a new CA root certificate. > and they can no longer run puppet because the puppet master's CA cert expired They can reconfigure internal tools to use the new CA root certificate, or rather one of the signed intermediate certificates. > and they can't get a new one because no one has access. They can simply g…
Serious question... How do I build a system that grants access to a company role not a person? In other words, the CIO is fired, how does this system ensure that the new CIO can access it, and the old one no longer can?
If we tie it to the HR system, whoever admins that effectively has the keys to the kingdom. Same for Active Directory or any other technical solution.
Re: Twitter has an internal root CA problem
#60> Musk fired everyone with access to the private key to their internal root CA, The way forward is to generate a new CA root certificate. > and they can no longer run puppet because the puppet master's CA cert expired They can reconfigure internal tools to use the new CA root certificate, or rather one of the signed intermediate certificates. > and they can't get a new one because no one has access. They can simply g…
LOL, you realize all the PEOPLE you list as the PEOPLE who should be able to manage the keys to the kingdom are PEOPLE? Board of directors - fired on day one of Musk takeover, executive managers - many fired one day one by Musk as well, designated maintainers - for all we know they could have been fired in the purge or quit when Musk offered the 3 month severance.
All system require people to run.