Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

461–470 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#461

Earlier quoted context omitted.

I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.

It would be funny for Musk to share privately with friends. It is a deeply strange and inappropriate thing for the CEO of a billion dollar company to post publicly. Something is deeply wrong with Musk's decision making process.

It is funny. I think it's hilarious and the fact he shared it in public makes it even more so. It's one of his most liked tweets, so a lot of other people also thought it was funny. Comedy is subjective, but if a lot of people think something is funny then it is objectively funny. And it is perfectly appropriate for public figures to reveal something of themselves and have some fun in public. Puritans will always be offended by this kind of sex joke, or by the gays being allowed to marry or whatever, but we laugh at them too, they make it even funnier.

In my opinion what would be inappropriate for a billionaire CEO, and reveal deep flaws in decision making process, and the decision making process of any government official or governing body who subsequently partners with that CEO, is to fraternize with the likes of Epstein after he was convicted, e.g., https://www.nytimes.com/2019/10/12/business/jeffrey-epstein-.... But I guess morality is also subjective so people might disagree with me there too.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#462

Earlier quoted context omitted.

> Twitter grew 3x on the headcount front There were multiple executives making $10m/yr+ There were board members There were shareholders Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"? These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc. How can us commenters on HackerNews sit from our arm…

> How can us commenters on HackerNews sit from our armchair and say "ah, goofballs should've just not let headcount get so high!" The cliche HN comment on sites like Twitter (and many, many others, any time headcount comes up) has always been "why do they need so many people?" I've mostly dismissed it the same way I dismiss "I could build Uber in a weekend," but with every other tech giant laying people off, maybe I…

My first job out of Uni was a company that ran financial services, logistic fulfilment, and retail. They had at least a dozen large in-house retail brands, several large warehouses with automation that Amazon was probably behind at the time, and they also ran serviced some of the most well known brands in the country - from their website to customer support to logistics. They had delivery networks, networks of direct shipping, vast call centres with thousands of staff. We integrated with the banking system, with all sorts of external companies.

Our total headcount was under 10 000. Most of that was logistics and call centres (taking retail orders). Our IT dept was literally about 200 ish. All our office staff fit within a few floors of a medium sized building. IT was less than a single floor.

I’d say our software, whilst serving fewer users, was completing tasks orders of magnitude more complex than Twitter. We didn’t have much down time, in fact almost zero beyond physical damage to our networks and the once every few years disaster. Our critical support team was a handful of greybeards who spent much of the time playing solitaire. Code reviews were tight and access to production strictly controlled.

What we did have was a very lean culture, carefully managed over a long time. The company was over 100 years old.

I do not understand how, with far more sophisticated tech, companies like Twitter need so many staff. We could have run Twitter with a thousand folks, including sales. From an engineering standpoint a few hundred would do. And most of us weren’t great engineers. There were a few very experienced freelancers, though. We all worked 0900-1730, but we worked 7 full hours every single day, every minute of those 7 hours. No beanbags. Most of that time was sat quietly coding in a very cleverly designed open plan office (no cubicles) that was like a library.

Looking back, I think the key insight was that we were very cost driven and absolutely kept everything as simple as possible. We had all the usual project overrun problems, usually because things turned out to be more complex than previously imagined. But it baffles me when I hear engineering head count at places like Twitter. If you had given us 1000 engineers, we wouldn’t have known what to do with them.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#463

Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is: https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc... ) If this is the new town square, you can forward my mail to a cabin in the woods.

bold strategy on space karen’s part, marketing twitter as the next thing not to put your dick in

Re: Why Twitter didn’t go down: From a real Twitter SRE

#464

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

No matter if it was or not and for better or worse:

If Twitter survives this without any major harm it will have profound consequences for the whole software industry.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#465
post #307

Earlier quoted context omitted.

I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.

That post really did bring the church ladies out. Puritanism runs deep.

[deleted]

Re: Why Twitter didn’t go down: From a real Twitter SRE

#467

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

I expect it all to work better with Musk in charge. He knows how to make scalable software and he knows about performant teams.

That he’s not going to realise these totally obvious first order consequences people are raising seems unlikely.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#468

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wh…

Except in these firing sprees it's not competent employees that remain - those people are usually the first to jump ship because they have options and see where it's headed. It's not like they get raises for staying to get a part of the savings from the cuts - usually they get pay freezes.

Restructuring is done on whatever idea the new owner(s) have in plan - which could be equally disconnected from "real product".

I'm not that optimistic about Twitter long term - never was TBH but this Musk thing is turning to a shitshow and also exposes what it's like working for him in his other companies.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#469

Earlier quoted context omitted.

It's not goofballs. It's generally misaligned incentives. Managing a 10,000 org leads to better job prospects than a 1000 person org, than a 100 person org, than a pizza box team. Organizations tend to bloat. Random, rapid cuts might not be the fix here, but headcount was too high.

> Organizations tend to bloat. Say I'm an engineer. And I write code all day. And I get paid by a company to do it. And it blows up in production. And I tell my boss "code tends to blow up". Or I tell my boss "code tends to take longer than estimated to deliver". I wouldn't be given lots of promotions/bonuses with that outlook. We're talking about a Twitter CEO with $30m/yr+ in total compensation. There's no way his/…

> We're talking about a Twitter CEO with $30m/yr+ in total compensation.

Jack, who was once the CEO of Twitter, has said that they hired too many people too fast.

Elon, who is "paid" way more than $30 million/year to run several companies, also apparently thinks it was very bloated.

I'm not necessarily saying the billionaires are right and the millions are wrong. I'm just pointing out the clear counterexamples to your appeal to authority. I honestly don't understand why you would be so surprised to find out that millionaires also make plenty of mistakes. The Twitter stock price to a degree is one indicator that the Twitter executives were making mistakes.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#470

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

Twitter used to experience significant downtime compared to all other major platforms and one of the reason was its lack of redundancies across everything. Headcount is one such thing and it takes manpower to automate infrastructures as discussed in the post. Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usuall…

The headcount at WhatsApp in 2013 was somewhere between 50-100, at which time they were servicing approx 400m MAU, which is more than users than Twitter has been able to boast for most of their existence.

Coincidentally, in 2013 SpaceX was just starting to provide commerical launch capacity, at which point I think they too had Not surprised Elon Musk thinks he can run twitter with a skeleton crew.

Post reply on HN