Why Twitter didn’t go down: From a real Twitter SRE • Hacker News Board

Re: Why Twitter didn’t go down: From a real Twitter SRE

#841

Post by lightbendover » Tue, Nov 22, 2022, 2:21 PM UTC

donkeyd wrote: post #372

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

> and I can keep going

You focused mostly on additive bloat, there's also multiplicative bloat in the form of multiple teams focused on building separate versions of the same product to increase likelihood of success and empire building where leaders don't actually have a remit large enough to support the team size they have, but they have woven a narrative that defends the necessity nonetheless. Put everything together and teams are very easily 6x+ larger than they absolutely need to be to get a product into market.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#842

Post by mcosta » Tue, Nov 22, 2022, 2:21 PM UTC

tiborsaas wrote: post #445

acknorabotr wrote: post #278

Earlier quoted context omitted.

It's crazy to think a guy who builds reusable rockets thinks he can run a complex technology operation like Twitter.

Well, he's definitely not afraid of blowing up complex systems :)

Run fast, break things

Re: Why Twitter didn’t go down: From a real Twitter SRE

#843

Post by PunchTornado » Tue, Nov 22, 2022, 2:21 PM UTC

moonchrome wrote: post #468

rjzzleep wrote: post #429

Earlier quoted context omitted.

The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wh…

Except in these firing sprees it's not competent employees that remain - those people are usually the first to jump ship because they have options and see where it's headed. It's not like they get raises for staying to get a part of the savings from the cuts - usually they get pay freezes. Restructuring is done on whatever idea the new owner(s) have in plan - which could be equally disconnected from "real product". I…

You realise how annoying it is when you see people around you in the company making as much money as you and doing nothing useful to actually generate revenue. Data scientists that write a blog post about the most used hashtag or emoticon. Or PMs that we don't need...

I'm staying and I'm glad the teams are trimmed.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#844

Post by Tade0 » Tue, Nov 22, 2022, 2:23 PM UTC

mjr00 wrote: post #451

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

It's crazy to think about, but many people who use and build software today, including HN readers/commenters, are young enough to have only been exposed to the SaaS, cloud-first era, where software built with microservices deployed from CI/CD systems multiple times per day is just the way things are done. You're totally right; if you don't make changes to the software, it's unlikely to spontaneously stop working, esp…

I remember how it was over a decade ago and one hallmark of such systems was that they were easy to exploit.

My friend in college would just go into Wordpress admin panels and the like by using common exploits because nobody updated PHP on their VPSes back then.

As someone who spent most of their career to date as a front-end developer I learned that as long as they have the budget, stakeholders are insatiable. It's just that ten years ago most of their ideas were either technically not feasible or very expensive.

Nowadays browsers are much more capable, so the pressure to produce more features is much greater.

To our own peril, we can do much more now.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#845

Post by v0idzer0 » Tue, Nov 22, 2022, 2:28 PM UTC

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Of course it was bloat. This whole “twitter is going to crash and burn” thing is a weird fantasy. Most likely it will just be run more efficiently by far less people.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#846

Post by donkeyd » Tue, Nov 22, 2022, 2:29 PM UTC

recuter wrote: post #765

itsoktocry wrote: post #676

Earlier quoted context omitted.

> removing engineers won't instantly crash the product. It'll happen slowly It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.

Quoted post unavailable.

> The job cuts are clearly justified because of the extremely toxic "work culture"

I'm very curious in what way the stuff Elon is doing is not creating toxic work culture? >MinimumI'm not denying there may have been toxic work culture before, but I don't see how replacing it with another form of toxic work culture is to be celebrated.

Also, writing this:

> There is no active independent thought around this subject to be found. Only vitriol and bile.

And then spewing a bunch of vitriol and bile really distracts from any point you were trying to make:

> akin to witnessing a fat child throwing an epic tantrum laying flat on a mall floor

> Straight up Cartmenesque.

> clearly justified because of the extremely toxic "work culture" / cult

> because mommy didn't buy them an xbox

Re: Why Twitter didn’t go down: From a real Twitter SRE

#847

Post by lightbendover » Tue, Nov 22, 2022, 2:31 PM UTC

hef19898 wrote: post #726

Aeolun wrote: post #719

Earlier quoted context omitted.

> thought that Twitter would crash within days of the engineers being fired Because they work for companies where the product would fail within days of them being fired themselves.

That's what people keep telling themselves. Truth is, everybody is replaceable. And usually, loss of institutional knowledge takes a while to show real effects.

Having been on the side of thinking I was absolutely irreplaceable on a team, I was more or less proven right when I left and the team failed to deliver anything at all for about 10 months, even having to shut down existing products, and is now limping along.

The truth in retrospect is that it was my fault (and my upper leadership's) that I wasn't replaceable. I created a knowledge silo around myself since I wanted to move fast and figured I could prevent the team from being bogged down in complexity if I just handled it myself and while that worked in regards to delivering out-sized results for the available bandwidth, it also was a risk that materialized as described above. So while I do believe that everyone should be replaceable and it's their responsibility to be, it's not always the case and products can live and die by it.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#848

Post by bdcravens » Tue, Nov 22, 2022, 2:32 PM UTC

mgkimsal wrote: post #589

mjr00 wrote: post #451

Earlier quoted context omitted.

It's crazy to think about, but many people who use and build software today, including HN readers/commenters, are young enough to have only been exposed to the SaaS, cloud-first era, where software built with microservices deployed from CI/CD systems multiple times per day is just the way things are done. You're totally right; if you don't make changes to the software, it's unlikely to spontaneously stop working, esp…

The other side of that is browsers. Even if you don’t change your code, the platform people are running your code in changes, automatically in many cases. New JS or CSS behavior in next safari or chrome? You need to patch/push to accommodate running environments that are outside your control.

Sounds like a good argument for minimal Javascript.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#849

Post by trashtester » Tue, Nov 22, 2022, 2:32 PM UTC

entropi wrote: post #541

rjzzleep wrote: post #429

Earlier quoted context omitted.

The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wh…

> The reality is most huge companies are majority bloat. This is true, and in my opinion, true for a reason. And that reason is not "most huge companies are dumb", as opposed to what Musk's cult seem to believe. The reality is, measuring what exactly is "bloat" and precisely cutting that bloat is extremely difficult and firing more than half of your workforce is probably like using a warhammer to do brain surgery.

Well, your alias actually refers to that process. Large organizations gain entropy over time. And entropy is precisly the inability to describe some structure at a micro/nano-level.

But while fighting the entropy can be hard, not doing so is almost certain to be lethal.

If an organism is infected by gangrene or cancer (the more extreme forms of entropy for a body), it may be more realistic to cut away whole body parts than to treat it in place, even if there is risk of sudden death.

It seems to em that this is what Musk is trying. Either Twitter will be gone within a year, or they may very well be sustainably profitable within 2-3 years.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#850

Post by btown » Tue, Nov 22, 2022, 2:33 PM UTC

resonious wrote: post #440

donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

> how does it persuade you of that? By flowing from many people think it's bloat - I'll tell you what's really going on to tiny team of 1~3 built whole infra for critical component . I'm not really trying to make commentary on whether or not Twitter engineering was bloat, or whether or not I think it'll hit problems in the future. Just commenting on the fact that the article broke my expectations a little bit as a re…

https://twitter.com/atax1a/status/1594880931042824192 provides a bit more context and nuance (albeit with a sardonic tone, but it's hard to blame someone for being sardonic who saw their entire professional community fired).

There's no doubt that OP built a great and stable automation layer on top of Mesos for caching workloads. But there are numerous other types of workloads on top of Mesos (including, I presume mission-critical database deployments that need well-disciplined draining protocols to shift between nodes), as well as administrative needs for the Mesos-to-infrastructure level, and things running on bare metal below the Mesos level. These things all needed dedicated SREs, and the absence of these SREs could result in a scenario like the one mentioned in the Twitter thread I linked - two obscure mutually-dependent components expire and cannot be re-provisioned using documented tools.

I also think an important meta-point is that when Twitter was bringing in substantial revenue from advertising, every minute of downtime would have significant costs - costs that could make it easily worthwhile to "over-provision" SRE talent. With advertisers pausing engagement, perhaps Twitter loses less money from a day-long outage than it would save having the right talent to turn a day-long outage into a minutes-long outage.

Twitter is only judged by its profitability (namely, Musk's ability to service debt without selling more Tesla stock than he already has), while most other tech companies (both public and private) are judged by both profitability and revenue growth. If you want both, larger SRE teams, to say nothing of feature development and regulatory compliance teams, start to make a lot more sense.