Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

441–450 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#441

Post by kybernetyk »

swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

>What exactly where these thousands of employees actually doing

They had wine on tap.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#443

Post by oxfordmale »

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Let's do a thought experiment and see what functions aren't needed to keep the light on for 30 days:

1) HR 2) Legal 3) Sales 4) Marketing 5) Payroll 6) Admin staff 7) Most of Engineering, other than the bare minimum of L1/L2/3 support.

As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#444

Post by yunohn »

rjzzleep wrote: post #429
poulsbohemian wrote: post #87

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wh…

> I mean wholly crap, this reminds me of old German industry where people retire in the place they started.

This is honestly uncalled for. Job hopping every 2-3yrs should not be an expected task.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#445

Post by tiborsaas »

acknorabotr wrote: post #278
ulkesh wrote: post #27

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…

It's crazy to think a guy who builds reusable rockets thinks he can run a complex technology operation like Twitter.

Well, he's definitely not afraid of blowing up complex systems :)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#446

Post by AndyMcConachie »

TranquilMarmot wrote: post #375
drdrey wrote: post #162

Earlier quoted context omitted.

Another thing we noticed at Netflix was that after services didn’t get pushed for a while (weeks), performance started degrading because of things like undiscovered memory leaks, threads leaks, disks filling up. You wouldn’t notice during normal operations because of regular autoscaling and code pushes, but code freezes tended to reveal these issues.

We used to have a horribly written node process that was running in a Mesos cluster (using Marathon). It had a memory leak and would start to fill up memory after about a week of running, depending on what customers were doing and if they were hitting it enough. The solution, rather than investing time in fixing the memory leak, was to add a cron job that would kill/reset the process every three days. This was easier…

"Have you tried turning it off and on again?"

Re: Why Twitter didn’t go down: From a real Twitter SRE

#447

Post by bergenty »

taolegal wrote: post #41
philistine wrote: post #33

Earlier quoted context omitted.

And Twitter's problem are nowhere near technological. The site needed to make more money, not reengineer the whole thing while advertisers are fleeing because Trump is back on on a whim!

So should the platform be guided by advertisers? Especially one that’s apparently the de facto public square?

In this case yes because you need to be profitable.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#448

Post by bjarneh »

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> I was partly expecting the rest of the article to explain to my why exactly it wasn't just bloat

Same here. I guess his header was on point in why Twitter is still up; but I was also interested in hearing about why Twitter actually needs all those people. If it can be run with 50-80% of the staff gone, that does sound like some bloat at least.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#449

Post by cm2187 »

twblalock wrote: post #409
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

It’s easy to think it’s bloat at a steady state. When something important goes down and nobody knows how to fix it, it looks different.

That being said it's not like twitter is a massively complex product with lots of different features. I can imagine you could keep it running with a skeletton team. Liasing with ads buyers excepted.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#450

Post by octodog »

powerapple wrote: post #391
summerlight wrote: post #348

Earlier quoted context omitted.

Twitter used to experience significant downtime compared to all other major platforms and one of the reason was its lack of redundancies across everything. Headcount is one such thing and it takes manpower to automate infrastructures as discussed in the post. Sure, you can run the platform with 1/10 headcount with significantly degraded user experiences (say ~98%). This is not a problem for startups but people usuall…

I think the opposite. Many softwares at its best when the team was small. Software companies have to hire many people because it needs to report growth to investors, headcount is one of the measurement of growth. It is not necessarily good for the product, actually many times, it hurts the product, but overall it is good for the company, the company will enter new areas, can explore new things. What Twitter is doing…

> Software companies have to hire many people because it needs to report growth to investors, headcount is one of the measurement of growth.

I mean this is just wrong. Companies are always under pressure to cut costs (employees) and it is always talked about when quarterly results are posted. Look at how the market reacted to Facebook's latest results and then again what happened when they laid off thousands of staff.

Post reply on HN