Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter has an internal root CA problem

izzodlaw.com

Post reply on HN
Print view View original

31–40 of 79 posts

Re: Twitter has an internal root CA problem

#31

Post by brazzy »

ilyt wrote: post #15

Oh, I know that problem, we did change Puppet root CA due to mishap of one of the admins during updating to sha256 certs. But IIRC (it was long time ago) Puppet CA cert by default are issued for like 10-20 years, would be a bit weird if true. Also, old versions didn't had trust chain "just" root CA so puppet master would have to have key for that on disk anyway, proper "root CA + leaf CA for puppetmasters" have been…

> It would only be really problematic if they also lost SSH access to those machines using Puppet. If you have root access the fix is not exactly hard.

> But then they fired people that did had access so that might also be a problem

Oh my, wouldn't that be delicious...

Gotta wonder how you'd go about fixing that, though. Assuming that those people's access was also tied to their employment and irrevocably voided when they were fired: I guess it would depend on how well those machines are secured against attackers with access to the hardware.

Re: Twitter has an internal root CA problem

#32

Post by justsomeadvice0 »

walrus01 wrote: post #29
ilyt wrote: post #17

Earlier quoted context omitted.

Create new root CA, ssh to machine, remove old certs, re-add machine to Puppet, sign the new CSR on Puppet master, then it will download new root. Very simple operation... if you have working SSH access with root. If they don't, well...

If you don't have ssh access with root, hopefully you have access to something like the underlying hypervisor, to do the equivalent of "sudo xl console vmname" on a xen dom0 to get what is logically the same as a physical serial tty (or local vga+keyboard) console on the domU machine. Or the VMware esxi emulated graphical console, etc. Or if it's a bunch of bare metal machines, hopefully someone old-school in the org…

To my knowledge all modern DCs have out-of-band networks for this sort of thing that provide serial access to the BMC chip, nothing old school about that. Old school is having to submit a ticket to Jerry in the DC to walk the crash cart down to box 55AE, hook up a serial console, run diagnostics, and attach the output back to the ticket. You only have to deal with Jerry occasionally now, usually when the BMC or power rails fail.

Re: Twitter has an internal root CA problem

#33

Post by walrus01 »

justsomeadvice0 wrote: post #32
walrus01 wrote: post #29

Earlier quoted context omitted.

If you don't have ssh access with root, hopefully you have access to something like the underlying hypervisor, to do the equivalent of "sudo xl console vmname" on a xen dom0 to get what is logically the same as a physical serial tty (or local vga+keyboard) console on the domU machine. Or the VMware esxi emulated graphical console, etc. Or if it's a bunch of bare metal machines, hopefully someone old-school in the org…

To my knowledge all modern DCs have out-of-band networks for this sort of thing that provide serial access to the BMC chip, nothing old school about that. Old school is having to submit a ticket to Jerry in the DC to walk the crash cart down to box 55AE, hook up a serial console, run diagnostics, and attach the output back to the ticket. You only have to deal with Jerry occasionally now, usually when the BMC or power…

There's more than a few people who've decided the security risk of full console capable bmc is not acceptable - and if other fail over systems are engineered appropriately, not necessary at all. BMC/IPMI intentionally disabled/not connected to any network.

Anecdotally I have seen a number of low cost x86-64 pseudo blade setups similar to open compute platform design stuff which have no oob. If a unit fails it's pulled entirely and put in a work queue for someone to repair.

Re: Twitter has an internal root CA problem

#34

Post by brazzy »

walrus01 wrote: post #29
ilyt wrote: post #17

Earlier quoted context omitted.

Create new root CA, ssh to machine, remove old certs, re-add machine to Puppet, sign the new CSR on Puppet master, then it will download new root. Very simple operation... if you have working SSH access with root. If they don't, well...

If you don't have ssh access with root, hopefully you have access to something like the underlying hypervisor, to do the equivalent of "sudo xl console vmname" on a xen dom0 to get what is logically the same as a physical serial tty (or local vga+keyboard) console on the domU machine. Or the VMware esxi emulated graphical console, etc. Or if it's a bunch of bare metal machines, hopefully someone old-school in the org…

> Or if it's a bunch of bare metal machines, hopefully someone old-school in the organization thought to deploy 48/96-port rs232 console serial concentrators and wire them up to the db9 serial port on each physical server. And you didn't disable all local serial tty in your operating config.

In a hacker folklore story this would 100% be the solution. And for some reason they'd have to use an original VT100 that some greybeard had lovingly restored at home.

Re: Twitter has an internal root CA problem

#35

Post by threeseed »

justsomeadvice0 wrote: post #30
threeseed wrote: post #27

Earlier quoted context omitted.

According to this [1] Twitter has 500,000+ servers spread across DCs, GCP and AWS. Which if we assume only a team of your size remains then it would take 300+ days. That would mean no OS patches etc which would put them firmly in the crosshairs of the FTC. [1] https://twitter.com/d_feldman/status/1562265193249390593

If you are split-cloud under a homogenous puppet master without homogeneous break-glass SSH access (which would be crazy) then probably your best bet is to just re-kick the world. But the scaling factor for this sort of thing is most certainly not team size; it's "how many X servers can be down at the same time", which will increase with your number of servers. In any case I think the FTC is the least of twitter's co…

Not sure if it's still the case but last time I had co-located servers you could access the systems via OOB without needing to reboot them in single user mode.

If it's not the case then Twitter is definitely in far more trouble because according to past engineers at least a few of their services needed manual intervention on a full scale reboot. And losing quorum in a distributed system is never pretty.

Re: Twitter has an internal root CA problem

#36

Post by PeterisP »

concordDance wrote: post #26
walrus01 wrote: post #7

Earlier quoted context omitted.

There's also possibility of: "If this person hadn't been fired, they could use some other form of credentials within twitter's internal systems plus a passphrase they have memorized to login to the private-key-repository system where the credentials for the root CA are stored and retrieve them. But as they were fired abruptly they are not inclined to help Musk. And nobody has asked them".

Arent abrupt firings the norm in the USA? My company had layoffs last year and the US people were gone the same day.

Sure, but you do abrupt firings of whole teams only if you don't need what these teams do.

It is quite reasonable if a company's response plan for scenario "what if we intentionally shoot ourselves in the head" is "don't do that, why would we do that?".

Re: Twitter has an internal root CA problem

#37

Post by hef19898 »

jojobas wrote: post #25
ilyt wrote: post #20

Earlier quoted context omitted.

We have 7 racks and 3 people working in ops and built Puppet setup "right". It's not hard. And their setup was probably right too. Just that nobody plans for "bus hit our entire ops team"

More commonly you don't plan for it, you make sure the entire ops team are never on the same bus, same plane, preferably not even in the same city.

All fine and good. Until the new owner just fires everyone over night anyway. At which point, I guess, it is not the previous ops team's problem anymore.

Re: Twitter has an internal root CA problem

#38

Post by hayst4ck »

I think people who work in reliability see this type of thing as the real existential threat to twitter. It's unrealistic that a large infrastructure would fall over overnight, but what is very realistic is small problems being neglected until they become big problems, or multiple problems happening at the same time.

This alone is probably manageable, it might even be simple but painful to handle for 2-15 of twitters employees (pre-firing) with specialized knowledge. If 3 people knew the disaster recovery plan and they all got fired because they were so busy maintaining things and fighting fires that they failed to get good reviews by building things, well I wouldn't be surprised. Likewise the employees trusted with extreme disaster recovery mechanisms are not the poor souls on H1Bs who don't have the option of leaving easily, so the people trusted with access might have already jumped ship since they aren't being coerced into staying on board with a mad man.

The real existential threat is another problem compounding on top of this or a disastrous recovery effort. Auto-remediation systems could do something awful. A master database could fall over and a replica be promoted, but if that happens twice, 4 times? Without puppet to configure replacement machines appropriately, there could be a very real problem very quickly. Similarly, extremely powerful tools, like a root ssh key, might be taken out, but those keys do not have seat-belts and one command typed wrong could be catastrophic. Sometimes bigger disasters are made trying to fix smaller ones.

Puppet can be in the critical path of both recovery (via config change) and capacity.

Re: Twitter has an internal root CA problem

#39

Post by plorg »

donohoe wrote: post #11

Taking it with a pinch of salt, but this stuff does happen. I've received calls from past employers, usually when they migrate a site I worked on to a new CMS or platform. There is some critical service (AWS, CDN credentials, domain related) etc. that no one knows who has access... Happily those appear to get resolved... but this... yikes (if true)

In a possibly more pedestrian example, my organization needed a re-mailer service set up and found out that the IT worker previously tasked with administration for that service had the MFA set up for his personal phone. I think they eventually got a hold of him to coordinate transfer of credentials, but knowing him, there was a 50% chance he could have left the company on bad terms and would have made things quite a bit more difficult.
Post reply on HN