Why Twitter didn’t go down: From a real Twitter SRE
matthewtejo.substack.com
Why Twitter didn’t go down: From a real Twitter SRE
1–10 of 1001 posts
Re: Why Twitter didn’t go down: From a real Twitter SRE
#2Hey, sorry for the new account, i just like to try my best to keep my online identity separate. this for better or for worse has my real name on it. Hope this is interesting!
Re: Why Twitter didn’t go down: From a real Twitter SRE
#3Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#4Hey, sorry for the new account, i just like to try my best to keep my online identity separate. this for better or for worse has my real name on it. Hope this is interesting!
Nice work! Does Twitter runs their own DCs or hosted somewhere else?
Re: Why Twitter didn’t go down: From a real Twitter SRE
#5Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.
Right. It’s not like if all the engineers walk out it’s suddenly going to fall over. It’s that when an issue comes up there may not be the expertise on hand to fix it. So the remaining twitter engineers should expect a rough time over the coming months.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#6I don't think it's that crazy. Private Equity and Wall Street doesn't have pressure for resilience, it has pressure for churn. As long as revenue and spending... exists... things are fine. Ideally they are numbers that grow. This doesn't lend itself to clearing out a development backlog, or engineers doing the most important things, it lends itself to rapid iteration and justifications for the iterations.
So now that it is owned by someone that doesn't need churn and just needs to reduce cost, people can focus on discreet resilience factors, just like any small tightly held software operation. Many of Twitter's pet projects go away, the ad sales relationship have to get re-evaluated too, but the core consumer product that everyone sees can be made resilient and operate cheaper.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#7Hey, sorry for the new account, i just like to try my best to keep my online identity separate. this for better or for worse has my real name on it. Hope this is interesting!
Nice work! Does Twitter runs their own DCs or hosted somewhere else?
As of three years ago, Twitter had 2 to 3 data centers, and was moving some stuff to GCP. Not sure if current state of things.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#8I wonder how big Twitter's infra costs are (capex). Based on the article, they need to build capacity > 2x peak loaf to tolerate one of the DCs failing. But if they instead had 4 and still only tolerated 1 failure, they'd only need capacity > 1.33x peak load.
This doesn't account for the extra overhead associated with extra DCs, but it seems like there's opportunities for major effeciency wins.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#9No post body was provided.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#10I don't think it's that crazy. Private Equity and Wall Street doesn't have pressure for resilience, it has pressure for churn. As long as revenue and spending... exists... things are fine. Ideally they are numbers that grow. This doesn't lend itself to clearing out a development backlog, or engineers doing the most important things, it lends itself to rapid iteration and justifications for the iterations. So now that…
> Private Equity and Wall Street doesn't have pressure for resilience, it has pressure for churn. As long as revenue and spending... exists... things are fine. Ideally they are numbers that grow. This doesn't lend itself to clearing out a development backlog, or engineers doing the most important things, it lends itself to rapid iteration and justifications for the iterations.
Did we read the same article, because it seemed clear that all that resilience work was done when the company was public shareholder owned, not under the new private owner.