Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter has an internal root CA problem

izzodlaw.com

Post reply on HN
Print view View original

61–70 of 79 posts

Re: Twitter has an internal root CA problem

#61

Post by Volundr »

plorg wrote: post #39
donohoe wrote: post #11

Taking it with a pinch of salt, but this stuff does happen. I've received calls from past employers, usually when they migrate a site I worked on to a new CMS or platform. There is some critical service (AWS, CDN credentials, domain related) etc. that no one knows who has access... Happily those appear to get resolved... but this... yikes (if true)

In a possibly more pedestrian example, my organization needed a re-mailer service set up and found out that the IT worker previously tasked with administration for that service had the MFA set up for his personal phone. I think they eventually got a hold of him to coordinate transfer of credentials, but knowing him, there was a 50% chance he could have left the company on bad terms and would have made things quite a…

One of the first things I do when leaving a company is remove all credentials from my password manager. Sure they should disable my accounts, but on the off chance they don't I still want it clear I don't have access.

It doesn't have to be a departure on bad terms, if they needed my TOTP codes I can't help them. That secret is already gone.

Re: Twitter has an internal root CA problem

#63

Post by anecdotal1 »

foolfoolz wrote: post #58
parasense wrote: post #57

> Musk fired everyone with access to the private key to their internal root CA, The way forward is to generate a new CA root certificate. > and they can no longer run puppet because the puppet master's CA cert expired They can reconfigure internal tools to use the new CA root certificate, or rather one of the signed intermediate certificates. > and they can't get a new one because no one has access. They can simply g…

when building secure systems one of the key principles is assume someone will leak the private key. this is how we get to hsm maybe another one is assume you will lose access to the hsm. sure spinning up a new trust chain is annoying but it wouldn’t take that long to do. totally agree this post is overblown

spinning up a new trust chain is not so hard, but deploying that trust chain to thousands of servers around the world when your automation tool isn't available to do it with is really, really hard.

Re: Twitter has an internal root CA problem

#64

Post by ep103 »

hayst4ck wrote: post #38

I think people who work in reliability see this type of thing as the real existential threat to twitter. It's unrealistic that a large infrastructure would fall over overnight, but what is very realistic is small problems being neglected until they become big problems, or multiple problems happening at the same time. This alone is probably manageable, it might even be simple but painful to handle for 2-15 of twitters…

That's okay, Musk tweeted that Twitter needs a complete, green-field rewrite 5 days ago, I'm sure that will solve the problem.

Re: Twitter has an internal root CA problem

#65

Post by caltheon »

Volundr wrote: post #59
parasense wrote: post #57

> Musk fired everyone with access to the private key to their internal root CA, The way forward is to generate a new CA root certificate. > and they can no longer run puppet because the puppet master's CA cert expired They can reconfigure internal tools to use the new CA root certificate, or rather one of the signed intermediate certificates. > and they can't get a new one because no one has access. They can simply g…

Significant edits for clarity. Serious question... How do I build a system that grants access to a company role not a person? In other words, the CIO is fired, how does this system ensure that the new CIO can access it, and the old one no longer can? If we tie it to the HR system, whoever admins that effectively has the keys to the kingdom. Same for Active Directory or any other technical solution.

Something like the nuclear football is probably the only answer. Something very obvious that is transferred with the role

Re: Twitter has an internal root CA problem

#66

Post by Volundr »

caltheon wrote: post #65
Volundr wrote: post #59

Earlier quoted context omitted.

Significant edits for clarity. Serious question... How do I build a system that grants access to a company role not a person? In other words, the CIO is fired, how does this system ensure that the new CIO can access it, and the old one no longer can? If we tie it to the HR system, whoever admins that effectively has the keys to the kingdom. Same for Active Directory or any other technical solution.

Something like the nuclear football is probably the only answer. Something very obvious that is transferred with the role

You're probably right, though honestly I'm not sure that helps here either. If I'm the CIO and Musk walks in and tells me to get out, I'm not going to go to any pains to make sure he knows about the football. Sure I'll leave it there in my desk, where if someone knows of it's existence they can find it, but it probably just ends up going in the dumpster or with the desk when he sells it.

Re: Twitter has an internal root CA problem

#67

Post by karmakaze »

hayst4ck wrote: post #38

I think people who work in reliability see this type of thing as the real existential threat to twitter. It's unrealistic that a large infrastructure would fall over overnight, but what is very realistic is small problems being neglected until they become big problems, or multiple problems happening at the same time. This alone is probably manageable, it might even be simple but painful to handle for 2-15 of twitters…

Whenever I hear specifics about likely ways things could fail, I always see a plan. "Hey this all makes sense, lets focus on having these areas covered before they come to pass."

Same goes when someone lists all the reasons why a proposal isn't viable. "Great, so we'll address those and be golden then?" Often they list them as fact without considering (or the ability to imagine) that they could be made viable with additional effort.

Re: Twitter has an internal root CA problem

#68

Post by rsynnott »

VoidWhisperer wrote: post #51
jojobas wrote: post #25

Earlier quoted context omitted.

More commonly you don't plan for it, you make sure the entire ops team are never on the same bus, same plane, preferably not even in the same city.

Unfortunately even those precautions wouldn't save you from the 'holy shit basicially the entire ops team got fired out of nowhere' scenario..

_Nothing_ can save a company from sufficiently incompetent future management, ultimately.

Re: Twitter has an internal root CA problem

#69

Post by dragonwriter »

concordDance wrote: post #26
walrus01 wrote: post #7

Earlier quoted context omitted.

There's also possibility of: "If this person hadn't been fired, they could use some other form of credentials within twitter's internal systems plus a passphrase they have memorized to login to the private-key-repository system where the credentials for the root CA are stored and retrieve them. But as they were fired abruptly they are not inclined to help Musk. And nobody has asked them".

Arent abrupt firings the norm in the USA? My company had layoffs last year and the US people were gone the same day.

> Arent abrupt firings the norm in the USA?

Abrupt firings of everyone with critical access, primaries and backups, is not, because its suicide. (Also why critical access roles are vetted carefully, because you want to make sure there is a lower-than-normal chance you will need to fire any of them, since that’s how you minimize the chance of a situation where you’d want to fire enough of them to cause a critical situation.)

If you do need decide there’s a problem that requires you to fire those people, you find every way possible to delay firing some of them while you expand the set of people with that access (which may be only momentary, by compelling them to hand over credentials as part of the exit process, if you have confidence that you can do that successfully.)

Re: Twitter has an internal root CA problem

#70

Post by dragonwriter »

DrScientist wrote: post #3

If this is true - who knows - then it reflects rather badly on the people who were fired - as they didn't implement safeguard for a 'run over by a bus' scenario when they were in charge.

There’s “someone got run over by a bus outside of our control” and “The people in charge direct a bus to run over everyone covering a key function”. You don’t really plan for the latter scenario when you are in charge, instead, you just don’t direct a bus to do that. If your successor decides to do that, that’s…on them.
Post reply on HN