21–30 of 79 posts
Earlier quoted context omitted.
Maybe building it right cost 5x, and you have a budget for 1x. Sometime money is not unlimited even at FAANG
To add on, people forget that Twitter was never really FAANG. It not only wasn't profitable but had no monetization plan for years. I'm sure it paid off for all the investors who got Elon's money but even as a Facebook competitor they don't have Facebook money.
I'll take the rumor with a grain of salt, but can anyone unpack what the recovery plan would be for something like this? It would obviously be a big problem, but where would you even start?
Assuming they’ve still got access to the servers themselves via SSH, you’d start by issuing a new root CA cert for the Puppetmaster and putting that in place, then you’ve got to issue a new cert for every client and distributing those. It’s not impossible, but it’s also going to be a pain in the backside to do.
This is because the whole idea is that you have inaccessible, locked down Production servers that only Puppet (which is driven from a central, governed configuration management source) has authority to configure i.e. no SSH and no root access.
Thus leaving the only option being to physically visit each server at the datacenter and issue the commands.
[1] https://www.puppet.com/docs/puppet/5.5/ssl_regenerate_certif...
If this is true - who knows - then it reflects rather badly on the people who were fired - as they didn't implement safeguard for a 'run over by a bus' scenario when they were in charge.
Say you're planning, well, _anything_, and someone says "but in five years, a weird billionaire might buy the company and mismanage it to such an extent that your contingency plans don't work". There's a good argument that the proper response is that (a) that is largely the weird billionaire's problem and (b) that it is impossible to defend against an arbitrarily incompetent speculative future weird billionaire.
If someone takes a hammer to an electricity distribution board and electrocutes themselves, the normal response is not "well, that's the electrician's fault; they should have thought of that".
If true, this would "reflect rather badly" on exactly one person. But, y'know, it'll need to join a rather long queue of poorly reflecting things.
Earlier quoted context omitted.
Maybe building it right cost 5x, and you have a budget for 1x. Sometime money is not unlimited even at FAANG
We have 7 racks and 3 people working in ops and built Puppet setup "right". It's not hard. And their setup was probably right too. Just that nobody plans for "bus hit our entire ops team"
Earlier quoted context omitted.
There's "run over by a bus" and "90% of the company got ran over by a bus" scenarios. The second one is rarely worth implementing.
There's also possibility of: "If this person hadn't been fired, they could use some other form of credentials within twitter's internal systems plus a passphrase they have memorized to login to the private-key-repository system where the credentials for the root CA are stored and retrieve them. But as they were fired abruptly they are not inclined to help Musk. And nobody has asked them".
My company had layoffs last year and the US people were gone the same day.
Earlier quoted context omitted.
Assuming they’ve still got access to the servers themselves via SSH, you’d start by issuing a new root CA cert for the Puppetmaster and putting that in place, then you’ve got to issue a new cert for every client and distributing those. It’s not impossible, but it’s also going to be a pain in the backside to do.
Been there before, we did exactly this; except over OOB+reboot-into-single-user (because SELinux). Took us a few days (~5k servers) but managed to get out of it with no public-facing downtime. The other way would have just been to rekick the world one box at a time. A number of integration tests were added after that disaster :)
Which if we assume only a team of your size remains then it would take 300+ days.
That would mean no OS patches etc which would put them firmly in the crosshairs of the FTC.
[1] https://twitter.com/d_feldman/status/1562265193249390593
I'll take the rumor with a grain of salt, but can anyone unpack what the recovery plan would be for something like this? It would obviously be a big problem, but where would you even start?
Circular dependencies can absolutely wreck you. For example, puppet could configure sudoers, and without puppet config being applied people who would normally expect access might not have it. So now you have to find a privileged ssh key for un-configured machines.
I would be surprised if twitter did not have a physical vault with a USB drive with a root SSH key on it. With that you can do just about everything.
I would be most terrified of machine churn. Auto-remediation systems or elastic capacity systems can result in lost capacity that can't come back until the configuration problem is resolved.
I'll take the rumor with a grain of salt, but can anyone unpack what the recovery plan would be for something like this? It would obviously be a big problem, but where would you even start?
Create new root CA, ssh to machine, remove old certs, re-add machine to Puppet, sign the new CSR on Puppet master, then it will download new root. Very simple operation... if you have working SSH access with root. If they don't, well...
Or the VMware esxi emulated graphical console, etc.
Or if it's a bunch of bare metal machines, hopefully someone old-school in the organization thought to deploy 48/96-port rs232 console serial concentrators and wire them up to the db9 serial port on each physical server. And you didn't disable all local serial tty in your operating config.
Earlier quoted context omitted.
Been there before, we did exactly this; except over OOB+reboot-into-single-user (because SELinux). Took us a few days (~5k servers) but managed to get out of it with no public-facing downtime. The other way would have just been to rekick the world one box at a time. A number of integration tests were added after that disaster :)
According to this [1] Twitter has 500,000+ servers spread across DCs, GCP and AWS. Which if we assume only a team of your size remains then it would take 300+ days. That would mean no OS patches etc which would put them firmly in the crosshairs of the FTC. [1] https://twitter.com/d_feldman/status/1562265193249390593