Why would twitter go down due to staff layoffs? The stuff I build would keep running for years without my involvement. Do you mean that other systems are hand cranked?
Why Twitter didn’t go down: From a real Twitter SRE
421–430 of 1001 posts
Re: Why Twitter didn’t go down: From a real Twitter SRE
#422Earlier quoted context omitted.
Another thing we noticed at Netflix was that after services didn’t get pushed for a while (weeks), performance started degrading because of things like undiscovered memory leaks, threads leaks, disks filling up. You wouldn’t notice during normal operations because of regular autoscaling and code pushes, but code freezes tended to reveal these issues.
Agreed, one of the craziest bugs I had to deal with was we had a distributed system using lots of infrastructure. Said distributed system started having trouble communicating with random nodes and sub-systems. I spent 3 hard days finding a Linux kernel bug where the ARP cache was not removing least recently accessed network addresses. Normally, this wouldn't be a big deal for a typical network because few networks wo…
Yeah, network scaling bugs are the most fun. The one I liked the most was when after expanding a pool of servers, they started to lose connectivity for a few minutes and then come back a minute or so later as if nothing happened.
Turns out we accidentally stretched one server VLAN too wide, to roughly 600 devices within one VLAN within one switch. The servers had more-or-less all-to-all traffic, and that was enough to generate so many ARP requests and replies that the switch's supervisor policer started dropping them at random, and after ten failed retries for one server the switch just gave up and dropped it from the ARP table.
Of course the control plane policer is global for the switch, so every device connected to the switch was susceptible, not just the ones in the overextended VLAN.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#423Earlier quoted context omitted.
Because politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own. And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."
It's actually scary how many people, even engineers, put their reputation on the line saying Twitter wouldn't survive the weekend. It wasn't just Twitter employees. It's like a mass psychosis of some kind. It comes off as a kind of desperation, as though they need Elon to fail. Why? What's driving that response?
> Why? What's driving that response?
Experience.
Many of us have worked at companies where there is a lot of duct tape holding things together and when you let go of entire teams (not just a large percentage) then it isn't unreasonable to be pessimistic. Especially when you know that in order to fix problem A you need to take B, D, E, C corrective actions in that order. And you learnt that through years of things going wrong.
More so at companies like Twitter where they never really reach a steady state. You constantly have large fluctuations in system stress e.g. World Cup, Trump rejoining etc.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#424I know most of my comment is going to be redundant, but I need to say it in order to point to it an year or two from now as an "I told you so!" Twitter is going to be here for a long long time. It's not going to suddenly shut down, it's not going to slowly decline, there is not going to be mass abandonment. Elon might be cocky, but at the end of the day he is a successful businessman. He didn't just sent out that loy…
What kind of woke ideology exactly? Sorry not familiar the current climate in USA.
Definition of woke: alert to injustice in society, especially racism
Somehow, some folks are now saying this is bad.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#425I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…
Quoted post unavailable.
He also doesn't make decisions like this from monologues in his head - he surrounds himself with people having deep expertise and makes bold decisions based on feedback; if something doesn't make sense, they'll stop doing it the next morning. In "normal" corp not only it'll be there 3 years from now, it'll also grow fat around it and spawn other idiotic initiatives with little to no (negative?) value.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#426Earlier quoted context omitted.
> absolutely bring a site down over time, is expired certs From today's Casey Newton's newsletter: In early December, a number of Twitter’s security certificates are set to expire — particularly those that power various back-end functions of the site. (“Certs,” as they are usually called, serve to reassure users that the website they are visiting is authentic. Without proper certs, a modern web browser will refuse to…
I can imagine both cases being true, that the renewal process is automated and that certs won't get renewed because institutional knowledge has left the door. Where I'm at, service-to-service TLS certificates (the bulk of our certs) are automatically rotated by our deploy systems. But there are always the edge cases: the certificates manually created a long time ago (predating any standardized monitoring systems) wit…
> certs won't get renewed because institutional knowledge has left the door.
The parallel reality where you need to be a veteran SRE with an MIT degree to operate the arcane tool 'certbot'.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#427Earlier quoted context omitted.
Quoted post unavailable.
Your reply is indistinguishable from a religious person exclaiming that "God is omniscient and moves in mysterious ways!" Stop worshipping billionaires.
You could say that if he was new to the party.
But he did change the world with his companies already, didn't he?
Re: Why Twitter didn’t go down: From a real Twitter SRE
#428Re: Why Twitter didn’t go down: From a real Twitter SRE
#429I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…
The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wholly unqualified interviewers, but they now also have to handle nightmare features that were built by people completely disconnected from the other side.
I'm not a friend of Elon's, but outside of the flashiness of the whole thing, I don't think his firing spree was wholly unwarranted.
The other day I saw a video of a bunch of people at twitter leaving that have been there for a decade or so. I mean wholly crap, this reminds me of old German industry where people retire in the place they started.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#430Earlier quoted context omitted.
Quoted post unavailable.
Your reply is indistinguishable from a religious person exclaiming that "God is omniscient and moves in mysterious ways!" Stop worshipping billionaires.
This is a ridiculous take and you are not really adding anything to the discussion.
Musk has successfully run companies in the past and is running two extremely successful companies right now.