Not sure when OP left, but Twitter has 3 datacenters now.
Over the summer, Yeah there is three but I didn't think the 3rd was really running yet. It was kind of just there with some small things dripping in.
Why Twitter didn’t go down: From a real Twitter SRE
111–120 of 1001 posts
Re: Why Twitter didn’t go down: From a real Twitter SRE
#112Earlier quoted context omitted.
software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…
Spot on. Absolutely hate this attitude that software sitting there just gathers wear and tear as if it's a mechanical device. Software is written with a particular target platform in mind: x86, ARM, Nvidia GPUs, FPGA soft-processor etc. If the hardware you are running on doesn't change, your software should still function. If the specs of that target platform don't change, your software should still function. If the…
Complex software has complex failure modes though.
An application running on a single platform, self-contained and with some basic failovers such as redundancy (2+ machines running the same application), etc. should have ridiculous uptimes.
A distributed and complex system with interdependent components, under variable load, with different capacities for subsystems running across some thousands of machines will, inevitably, encounter some unforeseen state which degrades the system as a whole. It can be a small component that cascades a failure mode in an unexpected way or can be a big component failing spectacularly due to a bug, or race condition, or a multitude of other issues that are not entirely predictable and guarded against at the time of writing such software.
The latter is what has "wear and tear", it's not one software, it's a whole system of software communicating with each other in varying states of hardware decay, you can design and build it to be resilient against a multitude of predictable issues but you can never expect that it will run perfectly fine unattended.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#113Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.
software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…
It’s possible to make things that last if you are in total control of the whole stack, including hardware.
Embedded systems that still do their job after 30 years do exist but they live in isolation in a specific and controlled environment, and are built for a limited, unchanging task.
On the other hand, complex web software is build on layer upon layers that are not in Twitter’s complete control.
Hardware change regularly, requiring changes at the lower levels of an OS, inducing potential changes in behaviour, performance, which require adaptation as a consequence.
And that’s before considering security, eternally moving goalposts. Not just at the OS or network level, but also at the business level.
Twitter and al are not living in a locked down context, they live in the messy world of human interactions and that alone requires constant tweaking.
So yes, a binary is more like a mathematical construct and by itself it won’t rot, but if the world around that binary changes, you need to change the binary as well, and for that you need maintenance. The amount required depends on the complexity, brittleness and how well your stack is engineered, but implying it’s a con is a bit extreme.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#114I am mostly curious about the Mesos layer itself. Mesos is dead. So you need in-house expertise to patch it without being able to leverage community knowledge. Does Twitter retain enough people to manage Mesos?
I was surprised to hear that a cutting edge social media app uses Mesos. Why did they choose that over other options?
Re: Why Twitter didn’t go down: From a real Twitter SRE
#115I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.
This is the real question? Your question has nothing to do with the blog post and if you take a look around, what Twitter did was literally done across the entire industry, hence all the layoffs recently. There was a hiring glut to take advantage of cheap capital during COVID recovery. The capital has dried up, glut has ended and a lot of people lost their jobs. Why is that so hard to see? None of this is unique in a…
A bunch of people just got axed from Twitter because covid cash dried up? Shit I thought it was because Elon took over and fired anyone who refused to work at the office instead of at home.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#116I am mostly curious about the Mesos layer itself. Mesos is dead. So you need in-house expertise to patch it without being able to leverage community knowledge. Does Twitter retain enough people to manage Mesos?
The company was moving off it. I wish I could find a twitter eng blog about it. Interesting there's a bunch of not directly from twitter sources about the decision though.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#117Earlier quoted context omitted.
> Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough. This seems to be more true of Mastodon than Twitter. I can't imagine any self hosted Mastodon instance staying up longer than twitter.
why? a single app running on a single server is several orders of magnitude more resilient than a spaghetti clusterfuck of services upon services. twitter could be brought down by a single expired certificate
most of the mastodon instances I saw were:
- ghost towns with little to no one there.
- didn't open registrations.
- completely dead with ssl invalid certificates with expired domains.
So you would have to keep moving to another mastodon instance (if you're lucky) or try and run your own instance and join the many instances with the three issues above.
There is no monetary incentive to keep a mastodon instance running and we both know that begging for donations doesn't scale.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#118I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.
It had to take a bunch of them to wreck the UI.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#119I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.
If the 3x headcount increase really did add no value, there are still about 1/3rd profitable employees there now. In fact giant layoffs tend to cut the best people first because they are the ones who feel comfortable walking. The people that are the last to go are the ones who are very entrenched in the organization and who don't estimate their chances outside of it highly, and that's the exact description of who Elo…
> In fact giant layoffs tend to cut the best people first because they are the ones who feel comfortable walking.
This is more true when the layoffs happen because the company’s situation deteriorates. If the company cuts jobs because revenues fall and products fail, better employees are indeed more likely to move to greener pastures before mediocre ones do. If, however, the company prospects improve, rather than worsen, this is no longer the case.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#120Earlier quoted context omitted.
The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…
I have a piece of Rust software that has not gone down in its entire lifetime.
Me too. It’s a hello world web service I wrote while reading a tutorial after work this afternoon.