11–20 of 79 posts
I've received calls from past employers, usually when they migrate a site I worked on to a new CMS or platform. There is some critical service (AWS, CDN credentials, domain related) etc. that no one knows who has access... Happily those appear to get resolved... but this... yikes (if true)
If this is true - who knows - then it reflects rather badly on the people who were fired - as they didn't implement safeguard for a 'run over by a bus' scenario when they were in charge.
Maybe building it right cost 5x, and you have a budget for 1x. Sometime money is not unlimited even at FAANG
If this is true - who knows - then it reflects rather badly on the people who were fired - as they didn't implement safeguard for a 'run over by a bus' scenario when they were in charge.
It would only be really problematic if they also lost SSH access to those machines using Puppet. If you have root access the fix is not exactly hard.
But then they fired people that did had access so that might also be a problem
We made sure all of our machines can be accesses both by Puppet and by SSH kinda for that reason; we had both accidents of someone fucking up Puppet, and someone fucking up SSH config rendering machines un-loggable (the lessons were learned and etched in stone).
So really, depending on who has access to what, it can be anything from "just pipe list of hosts to few ssh commands fixing it" to "get access manually to the server and change stuff, or redeploy machine from scratch". Again, assuming muski boy didn't fire wrong people
I'll take the rumor with a grain of salt, but can anyone unpack what the recovery plan would be for something like this? It would obviously be a big problem, but where would you even start?
Assuming they’ve still got access to the servers themselves via SSH, you’d start by issuing a new root CA cert for the Puppetmaster and putting that in place, then you’ve got to issue a new cert for every client and distributing those. It’s not impossible, but it’s also going to be a pain in the backside to do.
I'll take the rumor with a grain of salt, but can anyone unpack what the recovery plan would be for something like this? It would obviously be a big problem, but where would you even start?
Very simple operation... if you have working SSH access with root. If they don't, well...
Taking it with a pinch of salt, but this stuff does happen. I've received calls from past employers, usually when they migrate a site I worked on to a new CMS or platform. There is some critical service (AWS, CDN credentials, domain related) etc. that no one knows who has access... Happily those appear to get resolved... but this... yikes (if true)
But, well, if you fuck up your CM...
If this is true - who knows - then it reflects rather badly on the people who were fired - as they didn't implement safeguard for a 'run over by a bus' scenario when they were in charge.
There's "run over by a bus" and "90% of the company got ran over by a bus" scenarios. The second one is rarely worth implementing.
If this is true - who knows - then it reflects rather badly on the people who were fired - as they didn't implement safeguard for a 'run over by a bus' scenario when they were in charge.
Maybe building it right cost 5x, and you have a budget for 1x. Sometime money is not unlimited even at FAANG
Just that nobody plans for "bus hit our entire ops team"