Earlier quoted context omitted.
haha I'll go fix that. sorry :(
lol now i feel bad making fun
lol dont worry, nothing compared to how bad the anonymous internet can be anyway!
Why Twitter didn’t go down: From a real Twitter SRE
251–260 of 1001 posts
Re: Why Twitter didn’t go down: From a real Twitter SRE
#252Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.
software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…
Computation is literally bound by entropy. Math has no such limitations unless you explicitly define them.
I thoroughly recommend researching entropy as it regards to e.g. information theory, systems engineering and even (perhaps especially) to machine learning.
Computation is ultimately about what we can compute _in this universe_ and the forward flow of time is an emergent property from the universe’s innate entropic guarantees.
Time is “pre-sorted” for us thanks to entropy, enabling us to define algorithmic complexities over the time domain in the first place.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#253I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.
This is the real question? Your question has nothing to do with the blog post and if you take a look around, what Twitter did was literally done across the entire industry, hence all the layoffs recently. There was a hiring glut to take advantage of cheap capital during COVID recovery. The capital has dried up, glut has ended and a lot of people lost their jobs. Why is that so hard to see? None of this is unique in a…
No post body was provided.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#254I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.
If the 3x headcount increase really did add no value, there are still about 1/3rd profitable employees there now. In fact giant layoffs tend to cut the best people first because they are the ones who feel comfortable walking. The people that are the last to go are the ones who are very entrenched in the organization and who don't estimate their chances outside of it highly, and that's the exact description of who Elo…
I've found the opposite. I almost never see low performing employees fired outside of a mass layoff. In every layoffs I've seen 10x as many people were fired as quit. So you lose a bunch of low performers involuntarily, and a few top performers both voluntarily and involuntarily and that leads to the average quality improve.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#255Earlier quoted context omitted.
Spot on. Absolutely hate this attitude that software sitting there just gathers wear and tear as if it's a mechanical device. Software is written with a particular target platform in mind: x86, ARM, Nvidia GPUs, FPGA soft-processor etc. If the hardware you are running on doesn't change, your software should still function. If the specs of that target platform don't change, your software should still function. If the…
I have to reboot my phone weekly exactly because “stable” software isn’t. Even on a know platform in the absence of updates. The only good thing is that “turning it off and back on again” works. Usually.
Honestly, that's just a sign the software on your phone isn't stable. (most modern software isn't, to be fair)
Re: Why Twitter didn’t go down: From a real Twitter SRE
#256There's a pattern of diverging expectations here, one is the non-technical/naïve one, - Twitter is going to go down tomorrow and it's all over. RIP. The second is, - Twitter is going to experience a failure cascade over time. The third is, - It's all going to be fine. I suspect that the real question is, how many individual wires can break before the cable holding the suspended platform snaps? I am not that good of a…
Also anecdote, but twitter has historically been pretty unreliable for me. Without a reliable twitter systems status history pre-acquisition, the reports of failures, like the issues with the 2FA system, don't mean a whole lot.
Definitely this. I regularly see people complaining about ancient bugs as if they were new after the acquisition. Nope, you just are engaged enough to notice them now.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#257There's a pattern of diverging expectations here, one is the non-technical/naïve one, - Twitter is going to go down tomorrow and it's all over. RIP. The second is, - Twitter is going to experience a failure cascade over time. The third is, - It's all going to be fine. I suspect that the real question is, how many individual wires can break before the cable holding the suspended platform snaps? I am not that good of a…
Perhaps the failure mode is like the Tacoma Narrows Bridge, with Elon as the wind, the undulations his whipsaw changes of mind, and the employees the hapless cars being flung into the river.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#258Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is: https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc... ) If this is the new town square, you can forward my mail to a cabin in the woods.
I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.
[deleted]
Re: Why Twitter didn’t go down: From a real Twitter SRE
#259I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…
I did see a lot of politically-motivated salivating which I thought was woefully premature. That said, I see 2 outcomes now: 1. Generally, with large, complex systems like this, everything works, until it doesn't. All the big boys have major outages periodically. I just can't fathom how Twitter is going to handle the eventual certainty of a major outage when, as the author notes, in some cases there are teams that ha…
Generally agreed. However keep in mind that bankruptcy is a purely financial event and doesn't necessarily have to have any impact on operations.
Airlines are notoriously for going bankrupt regularly.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#260I will just wait for the bug the breaks the camels back here.
There will be a necessary breaking change that there will be no support for that will cascade into the downtime the media is demanding at the moment.