Earlier quoted context omitted.
He also sold all the smoke detectors.
Credit where it’s due though: he did supply a spare sink.
You should generally ignore sunk costs.
Why Twitter didn’t go down: From a real Twitter SRE
201–210 of 1001 posts
Re: Why Twitter didn’t go down: From a real Twitter SRE
#202Earlier quoted context omitted.
why? a single app running on a single server is several orders of magnitude more resilient than a spaghetti clusterfuck of services upon services. twitter could be brought down by a single expired certificate
most of the mastodon instances I saw were: - ghost towns with little to no one there. - didn't open registrations. - completely dead with ssl invalid certificates with expired domains. So you would have to keep moving to another mastodon instance (if you're lucky) or try and run your own instance and join the many instances with the three issues above. There is no monetary incentive to keep a mastodon instance runnin…
well, those are outright abandoned
Re: Why Twitter didn’t go down: From a real Twitter SRE
#203There's a pattern of diverging expectations here, one is the non-technical/naïve one, - Twitter is going to go down tomorrow and it's all over. RIP. The second is, - Twitter is going to experience a failure cascade over time. The third is, - It's all going to be fine. I suspect that the real question is, how many individual wires can break before the cable holding the suspended platform snaps? I am not that good of a…
It looks like direct messages may also be buggy at present, according to a couple of anecdotes on r/Twitter at: https://old.reddit.com/r/Twitter/comments/z1b4zc/cant_sendre...
Re: Why Twitter didn’t go down: From a real Twitter SRE
#204Earlier quoted context omitted.
It's not goofballs. It's generally misaligned incentives. Managing a 10,000 org leads to better job prospects than a 1000 person org, than a 100 person org, than a pizza box team. Organizations tend to bloat. Random, rapid cuts might not be the fix here, but headcount was too high.
> Organizations tend to bloat. Say I'm an engineer. And I write code all day. And I get paid by a company to do it. And it blows up in production. And I tell my boss "code tends to blow up". Or I tell my boss "code tends to take longer than estimated to deliver". I wouldn't be given lots of promotions/bonuses with that outlook. We're talking about a Twitter CEO with $30m/yr+ in total compensation. There's no way his/…
This is a appeal to imperfect authorities that make mistakes all the time. By that logic, Elon was able to convince banks to loan 14B to acquire and gut Twitter. There was no way the staff and shareholders of those organizations, much larger and more proven than Twitter could have been so complacent and ignorant to make such a mistake.
Elon himself is a CEO with a better track record than Parag, why is he not more legitimate if we are pretending leaders can't make mistakes
Re: Why Twitter didn’t go down: From a real Twitter SRE
#205I am mostly curious about the Mesos layer itself. Mesos is dead. So you need in-house expertise to patch it without being able to leverage community knowledge. Does Twitter retain enough people to manage Mesos?
The Mesos detail struck me as a risk for the reason you stated.
Also, the article seems to suggest Twitter only has two datacenters. That seems surprising for the global reach of the company. Perhaps there are other smaller datacenters that are not prepared to handle the entirety of the site’s traffic.
My current thinking is there’s time to figure out how to operate the current system before it runs into issues that would render it degraded for a prolonged period of time. I noticed TLS certs have already rotated for instance. That was my best guess for simple thing that could fail if managed poorly.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#206Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is: https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc... ) If this is the new town square, you can forward my mail to a cabin in the woods.
You do know that you can block Elon's account.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#207Earlier quoted context omitted.
The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…
> all software goes down and requires specialist intervention eventually Well, that’s it, isn’t it? How many software systems need to keep running for Twitter to remain more or less functional? If there are 10 critical systems that are running at four 9’s, you’d expect 3.6 hours of downtime a year, or about 90 days of uptime at a stretch if I have my math right. If there are 100 critical systems running at 3 9’s, you…
Yep, and when one of the SEVs rolls around that would have been small (say 5m of downtime fixed with a flag flip), it instead will have a nontrivial chance of escalating into a major multi-hour/multi-day outage without the right institutional knowledge.
I'd guesstimate that Twitter probably has dozens of services that are in the critical path of an average user interaction. It's hard to keep even logically optional dependencies truly optional in large scale systems involving many people.
However Twitter didn't die in the past when fail whales ruled its day, so they probably won't kill it now. It's just not that kind of business. (In contrast, a one hour outage had me directly apologizing to our largest customers on the phone). That said, Twitter can only be unstable and lack feature growth for so long before something else takes its place, so Musk is on a clock.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#208I know most of my comment is going to be redundant, but I need to say it in order to point to it an year or two from now as an "I told you so!" Twitter is going to be here for a long long time. It's not going to suddenly shut down, it's not going to slowly decline, there is not going to be mass abandonment. Elon might be cocky, but at the end of the day he is a successful businessman. He didn't just sent out that loy…
What kind of woke ideology exactly? Sorry not familiar the current climate in USA.
No post body was provided.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#209Earlier quoted context omitted.
To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...
Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to. So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building . But eventually people are going to start turning on stoves again… an…
This is correct, but also: a sudden decline in maintenance is a kind of change in its own right. Even automated processes have humans in the loop and manual sign-offs; there's always some cronjob or short-lived certificate somewhere that a human was dutifully maintaining.
Those things aren't going to fail any sooner than they would have anyways, but they're going to fail a lot harder due to the loss of institutional knowledge.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#210Tangent but I do hope that Musks trim down results in orgs that have less “executives” and a layered cake of a org structure, and more autonomous small teams that execute on shared overarching initiatives. I really don’t understand why so many tech companies have like 8 layers of engineering levels. If the argument is that you need more money so more levels, just have a bigger band. Don’t chase titles they don’t mean…
A lot of those upper layers and the bureaucracy which is seen as "bloat by the bottom layers comes about from compliance. The compliance burden grows as the business grows (e.g PCI, insurance and government requirements kick in at certain thresholds).
Sure engineering adding a new widget to the site might increase profits by 12%, but all that bureaucracy can prevent the company losing its payment provider or breaching a government regulation which might cause the company to close overnight. So if the stakes are +12% vs -100%, who is actually doing the most important work?
I don't disagree with your desire to work somewhere lean and task-focused, but I think it's almost impossible for that to happen anywhere but small(er) workplaces.