Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter has an internal root CA problem

izzodlaw.com

Post reply on HN
Print view View original

41–50 of 79 posts

Re: Twitter has an internal root CA problem

#41

Post by yuppie_scum »

8organicbits wrote: post #9

I'll take the rumor with a grain of salt, but can anyone unpack what the recovery plan would be for something like this? It would obviously be a big problem, but where would you even start?

Depends how disposable the individual servers are. I don't know specifics of the Twitter infra, but I would probably just issue a new cert and begin shooting and replacing the old servers. Hopefully the services are abstracted from the puppet cert and things like Redis and whatnot will safely reprovision and find their quorums.

Re: Twitter has an internal root CA problem

#42

Post by yuppie_scum »

ilyt wrote: post #17
8organicbits wrote: post #9

I'll take the rumor with a grain of salt, but can anyone unpack what the recovery plan would be for something like this? It would obviously be a big problem, but where would you even start?

Create new root CA, ssh to machine, remove old certs, re-add machine to Puppet, sign the new CSR on Puppet master, then it will download new root. Very simple operation... if you have working SSH access with root. If they don't, well...

If they're in the cloud, it's pretty straightforward to re-mount the drive somewhere else and replace the SSH keys.

Re: Twitter has an internal root CA problem

#43

Post by nunez »

walrus01 wrote: post #2

very possibly bullshit but huge if true: https://twitter.com/davidgerard/status/1634633886712954881

wouldn't put it past him since he only wanted "builders" and I bet he doesn't consider platform ops "builders" (even though they build tons of stuff; twitter's platform was basically a product in and of itself)

Re: Twitter has an internal root CA problem

#44

Post by rakoo »

threeseed wrote: post #27
justsomeadvice0 wrote: post #16

Earlier quoted context omitted.

Been there before, we did exactly this; except over OOB+reboot-into-single-user (because SELinux). Took us a few days (~5k servers) but managed to get out of it with no public-facing downtime. The other way would have just been to rekick the world one box at a time. A number of integration tests were added after that disaster :)

According to this [1] Twitter has 500,000+ servers spread across DCs, GCP and AWS. Which if we assume only a team of your size remains then it would take 300+ days. That would mean no OS patches etc which would put them firmly in the crosshairs of the FTC. [1] https://twitter.com/d_feldman/status/1562265193249390593

Interesting. Considering the number of MAU to be around 350 million, that's a bit fewer than 1000 persons per server. Of course it's not that simple, because not all servers are the same and more importantly not all users are the same, but it sounds like a bit on the low end.

Anecdotal point of data: infosec.exchange hosts 30k users on 7 servers (https://infosec.exchange/@jerry/109374478717918484). That's a 1:5 ratio. Again, not the same usage and performance requirements, but I find it interesting.

Re: Twitter has an internal root CA problem

#46

Post by orwin »

DoctorOW wrote: post #12
raizer88 wrote: post #5

Earlier quoted context omitted.

Maybe building it right cost 5x, and you have a budget for 1x. Sometime money is not unlimited even at FAANG

To add on, people forget that Twitter was never really FAANG. It not only wasn't profitable but had no monetization plan for years. I'm sure it paid off for all the investors who got Elon's money but even as a Facebook competitor they don't have Facebook money.

It was profitable in 2019 and 2020, and could have been in 2022 (the first semester was huge, probably because of all the stuff happening all over the world).

Re: Twitter has an internal root CA problem

#49

Post by tedivm »

plorg wrote: post #39
donohoe wrote: post #11

Taking it with a pinch of salt, but this stuff does happen. I've received calls from past employers, usually when they migrate a site I worked on to a new CMS or platform. There is some critical service (AWS, CDN credentials, domain related) etc. that no one knows who has access... Happily those appear to get resolved... but this... yikes (if true)

In a possibly more pedestrian example, my organization needed a re-mailer service set up and found out that the IT worker previously tasked with administration for that service had the MFA set up for his personal phone. I think they eventually got a hold of him to coordinate transfer of credentials, but knowing him, there was a 50% chance he could have left the company on bad terms and would have made things quite a…

I had something similar happen when I left a company, only I'm fairly consistent on deleting credentials to systems I'm not supposed to have access to. Fortunately it was for an internal service and nothing customer facing, so they were able to wipe and redeploy.

Re: Twitter has an internal root CA problem

#50

Post by ilyt »

justsomeadvice0 wrote: post #32
walrus01 wrote: post #29

Earlier quoted context omitted.

If you don't have ssh access with root, hopefully you have access to something like the underlying hypervisor, to do the equivalent of "sudo xl console vmname" on a xen dom0 to get what is logically the same as a physical serial tty (or local vga+keyboard) console on the domU machine. Or the VMware esxi emulated graphical console, etc. Or if it's a bunch of bare metal machines, hopefully someone old-school in the org…

To my knowledge all modern DCs have out-of-band networks for this sort of thing that provide serial access to the BMC chip, nothing old school about that. Old school is having to submit a ticket to Jerry in the DC to walk the crash cart down to box 55AE, hook up a serial console, run diagnostics, and attach the output back to the ticket. You only have to deal with Jerry occasionally now, usually when the BMC or power…

In both cases it's disruptive event as you have to reboot the machine to get into rescue mode (as you don't need the password)
Post reply on HN