Hey, sorry for the new account, i just like to try my best to keep my online identity separate. this for better or for worse has my real name on it. Hope this is interesting!
So you still work there? Or did you quit recently?
Why Twitter didn’t go down: From a real Twitter SRE
121–130 of 1001 posts
Re: Why Twitter didn’t go down: From a real Twitter SRE
#122Earlier quoted context omitted.
The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…
I had MySQL running on some bare metal for many years without a restart. I was terrified to update the kernel at that point, knowing that system disk had been running continuously for many years, and had no faith it would restart successfully. Finally got two new servers to replace these (with these new SSD things!) and after migration, sure enough, one of the old servers failed to boot.
Having single point of failure, and also not even knowing if it will even come up after reboot is a horrible way to run the service
Re: Why Twitter didn’t go down: From a real Twitter SRE
#123I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…
I did see a lot of politically-motivated salivating which I thought was woefully premature. That said, I see 2 outcomes now:
1. Generally, with large, complex systems like this, everything works, until it doesn't. All the big boys have major outages periodically. I just can't fathom how Twitter is going to handle the eventual certainty of a major outage when, as the author notes, in some cases there are teams that have 0 people left.
2. More than the technical issues, betting that Twitter will go bankrupt is the easiest bet one can make. Musk saddled Twitter with a shit ton of debt - even if things worked as they did before he had to cut tons of people due to the debt burden.
The issue I see is that #2 directly works against #1. Musk has said it will be lots of intense work adding new features to try to raise revenue. But making a ton of changes, probably with lots of shortcuts to get them out the door quickly, especially when so much institutional knowledge has walked out the door, will make keeping the site stable even that much harder.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#124Earlier quoted context omitted.
software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…
Moot in this case. Musk has gutted the work-force and instigated a wide range of changes. All bets are off when new code is being introduced.
Exhibit B: non-code policy changes have non-zero effects on your infrastructure. https://twitter.com/atax1a/status/1594905195737419777
> we don’t think so. the prod incident we heard about involved someone making an ill-advised choice to reactivate a large account, causing a huge load on the social-graph system, on the night before a prolonged high-traffic event.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#125Earlier quoted context omitted.
This is the real question? Your question has nothing to do with the blog post and if you take a look around, what Twitter did was literally done across the entire industry, hence all the layoffs recently. There was a hiring glut to take advantage of cheap capital during COVID recovery. The capital has dried up, glut has ended and a lot of people lost their jobs. Why is that so hard to see? None of this is unique in a…
A bunch of people just got axed from Twitter because covid cash dried up? Shit I thought it was because Elon took over and fired anyone who refused to work at the office instead of at home.
The question was why they were hired, not why they were fired...
Re: Why Twitter didn’t go down: From a real Twitter SRE
#126Earlier quoted context omitted.
If the 3x headcount increase really did add no value, there are still about 1/3rd profitable employees there now. In fact giant layoffs tend to cut the best people first because they are the ones who feel comfortable walking. The people that are the last to go are the ones who are very entrenched in the organization and who don't estimate their chances outside of it highly, and that's the exact description of who Elo…
> In fact giant layoffs tend to cut the best people first because they are the ones who feel comfortable walking. This is more true when the layoffs happen because the company’s situation deteriorates. If the company cuts jobs because revenues fall and products fail, better employees are indeed more likely to move to greener pastures before mediocre ones do. If, however, the company prospects improve, rather than wor…
Twitter has lost a lot of revenue recently and is now in a lot of debt, although that might not have had time to sink in. In real terms it is in a much worse place, and remember that betting on a success is something you'd do as an investor, not as a rank and file employee.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#127Earlier quoted context omitted.
software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…
Spot on. Absolutely hate this attitude that software sitting there just gathers wear and tear as if it's a mechanical device. Software is written with a particular target platform in mind: x86, ARM, Nvidia GPUs, FPGA soft-processor etc. If the hardware you are running on doesn't change, your software should still function. If the specs of that target platform don't change, your software should still function. If the…
I have to reboot my phone weekly exactly because “stable” software isn’t. Even on a know platform in the absence of updates.
The only good thing is that “turning it off and back on again” works. Usually.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#128Earlier quoted context omitted.
software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…
If software ran without side effects, perhaps. But it doesn't. Databases grow, files are uploaded, logs pile on, messages and events propagates and filesystems fill up. This is why entropy matters.
Exactly. Tiny memory leaks in seldom called functions can also cause slow degradation over time. People wonder why a simple restart seems to 'fix a boatload of problems' but this is often the reason why.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#129There's a pattern of diverging expectations here, one is the non-technical/naïve one,
- Twitter is going to go down tomorrow and it's all over. RIP.
- Twitter is going to experience a failure cascade over time.
- It's all going to be fine.
I am not that good of a developer, but watching Twitter I can't help but be reminded of Arecibo, except at a larger, more abstract scale. There was no single massive event that caused the failure, rather a series of factors and events, tiny cables breaking that eventually leads to a failure cascade that then causes the suspended platform to crash.
From what I can tell, in the past week or so,
- Twitter's copyright system failed
- Two Factor Authentication broke down (it seems to be back up?)
- (anecdata) Tweets have been loading sporadically for me and other people, sometimes we try to open a tweet and it says that it doesn't exist. Happens more frequently with new/recent tweets.
- (unconfirmed) Twitter's managed account backend is behaving "strangely." For e.g., "One of my campaign managers logged in last week and found all our paused creatives from the past 6 years had been reactivated." from https://www.teamblind.com/post/i-told-my-team-to-pause-our-750kmonth-twitter-ads-budget-last-week-4dnbo1Ft ———— Friends have told me other similar stories
I am asking this question because I don't know. And I'd like to develop a better mental model to understand what happens next.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#130From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running.
The biggest thing that brings down a site is changes. Typically code changes, but also schema/data changes, infra/network/config changes, etc. As long as nothing changes, and you don't run out of disk space (from logs for example), things stay working pretty much just fine. The trick is to design it to be as immutable and simple as possible.
There are other things that can bring a site down, like security issues, or bugs triggered by unusual states, too much traffic, etc. But generally speaking those things are rare and don't bring down an entire site.
The last thing off the top of my head that will absolutely bring a site down over time, is expired certs. If, for any reason at all, a cert fails to be regenerated (say, your etcd certs, or some weird one-off tool underpinning everything that somebody has to remember to regen every 360 days), they will expire, and it will be a very fun day at the office. Over a long enough period of time, your web server's TLS version will be obsoleted in new browser versions, and nobody will be able to load it.