Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

331–340 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#331
post #79

Earlier quoted context omitted.

> Twitter added 1.6M daily active users this past week, another all-time high https://twitter.com/elonmusk/status/1594865247323660290

Louis Rossmann said something like "there's a difference between people showing up to laugh at you vs. laugh with you."

No post body was provided.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#332
post #284

Earlier quoted context omitted.

Because politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own. And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."

It's actually scary how many people, even engineers, put their reputation on the line saying Twitter wouldn't survive the weekend. It wasn't just Twitter employees. It's like a mass psychosis of some kind. It comes off as a kind of desperation, as though they need Elon to fail. Why? What's driving that response?

There are a lot of places where the systems would start to fall over if 70-80% of the team departed. Especially since a lot of folks left on bad terms and/or were suddenly terminated. It was the opposite of a smooth handover.

So it wasn't unreasonable to think that Twitter would begin experiencing problems.

I didn't think that Twitter was going to literally have an unrecoverable system crash and permanently shut its doors, but I thought we'd see some outages or partial breakage over time, which is basically what has happened albeit in an admirably mild way.

    It comes off as a kind of desperation, as 
    though they need Elon to fail.
You don't need to pathologize it, like it's some... deep weirdo psychological yearning. Some people think he's a jerk and wouldn't mind seeing him fall on his face!

If a person was staking some significant part of their emotions on their feelings for Musk, yeah, that'd be unhealthy.

But I think you are significantly overestimating the emotional weight behind 99.99% of the half-baked Twitter quick takes. It's okay to not like a guy!

Re: Why Twitter didn’t go down: From a real Twitter SRE

#333

Earlier quoted context omitted.

America is such a great country that a random person can just fecklessly blunder into creating a revolutionary electric car company and cluelessly blunder into creating a rocket company that is the envy of the world.

Automotive and aerospace are not that similar to social media. People buying into the vision of "get the planet off fossil fuels for transport" and "get this species to Mars" are probably willing to make sacrifices that people working on social media are not. It's the Halo Effect fallacy to think competence in one field automatically translates to another. Especially when the founder in question has displayed increas…

Automotive and aerospace are not similar to each other, either. I don't know any other outfit that was successful at both.

> It's the Halo Effect fallacy to think competence in one field automatically translates to another

I didn't say it was. I was responding the notion that Musk blundered into success at Tesla and SpaceX.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#334

There's a pattern of diverging expectations here, one is the non-technical/naïve one, - Twitter is going to go down tomorrow and it's all over. RIP. The second is, - Twitter is going to experience a failure cascade over time. The third is, - It's all going to be fine. I suspect that the real question is, how many individual wires can break before the cable holding the suspended platform snaps? I am not that good of a…

Also anecdote, but twitter has historically been pretty unreliable for me. Without a reliable twitter systems status history pre-acquisition, the reports of failures, like the issues with the 2FA system, don't mean a whole lot.

After the failwhale days, Twitter has been quite stable for me with the only exception being the live-refresh features on tabs of the website that have been open for days (which I don't think many websites would handle well).

There has been a serious degradation in the quality since the acquisition:

- Sporadically loading tweets - I could go on some tweets and refresh the page multiple times, with tweets fading in and out of existence showing "This tweet is unavailable"

- Tweets that quote tweets of accounts you have blocked behave weird in multiple ways. Sometimes it's just showing a "This tweet is unavailable" instead of "This tweet is from an account you have blocked", and a few interacting with them crashed my timeline on mobile, having to restart the app

- On a few occurences, every third tweet on the timeline was an ad

- 1-2 notifications from crypto spam bots reaching me every day. The same thing previously was filtered out quite reliably (I assume), since then it happened ~once ever 6 months

--------

And those are only the things I've seen personally. Yeah, they are no deal-breakers, and mostly sporadic failures, but it very much feels like a service that is degrading by the day.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#335

Earlier quoted context omitted.

It's not goofballs. It's generally misaligned incentives. Managing a 10,000 org leads to better job prospects than a 1000 person org, than a 100 person org, than a pizza box team. Organizations tend to bloat. Random, rapid cuts might not be the fix here, but headcount was too high.

> Organizations tend to bloat. Say I'm an engineer. And I write code all day. And I get paid by a company to do it. And it blows up in production. And I tell my boss "code tends to blow up". Or I tell my boss "code tends to take longer than estimated to deliver". I wouldn't be given lots of promotions/bonuses with that outlook. We're talking about a Twitter CEO with $30m/yr+ in total compensation. There's no way his/…

Nope, just misaligned incentives. It really can be this simple sometimes.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#336
post #246

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

Redundancy. Now the systems are stable but human workers either be sick, leave, or die eventually. Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them. The article wrote about achieving stability by the distributed system so an unexpected death of one rack doesn't affect the service availability. The same can be done for…

> Rising the pay has diminishing returns. You can't prevent workers leaving because of lost of interests, be sick or die by throwing more money at them.

People would absolutely be more engaged and more excited about their work if they were paid more. The only reason people work is literally for money…

Re: Why Twitter didn’t go down: From a real Twitter SRE

#337

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people beca…

This whole thing has been quite telling for me. I don’t use Twitter and am not a Musk fan, but seeing someone try something to move away from ad driven seemed like a good possibility for me. Then I saw how bitter and nasty a lot of the online communities I belonged to were. I mention that advertisers were of course mad, the stated goal is to reduce reliance on ad revenue. I was met with people attacking me as an idio…

>I mention that advertisers were of course mad, the stated goal is to reduce reliance on ad revenue

In the computer industry, there are some famous historical examples of companies that announced their new product before it was ready, people stopped buying the old one, and they went out of business before the new version was done.

Not quite the same business, but a bit reminiscent.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#338
post #290

automation is indeed key. IRC networks run with no obvious employees and salarys and been running fine since the dawn of the internet. i see Twitter as a centralised, walled IRC. the # was pinched from IRC if u remember.

An IRC network server can serve what, a few hundred thousands of concurrent users’ text messages on a single PC.

Twitter has around 200 million daily user, peaking up to a billion, running on their own server farm. They are not remotely comparable, especially if we note that read-write dbs are notoriously hard to scale.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#339

...because it's only been a week and it takes longer for things to rot badly?

servers don’t really rot if they don’t rely on 3rd party cloud software. Linux doesn’t upgrade itself. the web is very standards based. Hacker news has pretty much been running for decades on the same software hardware stack

I’m fairly sure HN had outages as well.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#340
> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat.

I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually persuade me that it was all bloat.

Post reply on HN