Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

521–530 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#521
Its possible to teach a horse to eat less and less each day. They just have this unfortunate tendency to die just after you succeed in weaning them off food altogether.

A horse, of course, is not same as a reduction in force.

Engineers don't eat hay, neither does a horse have a 401(k).

Re: Why Twitter didn’t go down: From a real Twitter SRE

#522

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes . MBA's will be studying this for years.

Did Musk fire the right people? Or did he slash based on obedience, mood, whim? In what situation would one want THE senior SRE for the cache systems (which are unique and affect twitter across the board) GONE? There's a reason why managers were trying to claw back some engineers and employees after the big layoff. Sure you chop off a few fingers and keep working.... but lose a thumb?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#523

Earlier quoted context omitted.

Let's do a thought experiment and see what functions aren't needed to keep the light on for 30 days: 1) HR 2) Legal 3) Sales 4) Marketing 5) Payroll 6) Admin staff 7) Most of Engineering, other than the bare minimum of L1/L2/3 support. As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend.

This is more like a car with just the bare essentials: an engine, steering wheel, brakes, and the like. No: radio, air-conditioning, seat padding, wipers, lights, radar, etc... Oh, and no maintenance. It'll drive ... for now. But that's it.

More like a racecar without maintenance. Heres an example checklist what they need: https://www.teamstradale.com/frequency-of-maintenance-keepin...

Re: Why Twitter didn’t go down: From a real Twitter SRE

#524

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

That's likely optimistic.

Maybe the user facing site, but that's just the tip of the iceberg.

There are plenty of internal/backend/restricted systems to support and/or monetize this part.

And that's not counting the huge number of support people & moderators needed.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#525

Earlier quoted context omitted.

But real world conditions can force code changes. For example, a region abandons daylight savings time or a court order on copyright infringement. Someone unqualified working a system they are unfamiliar with could blow it up. Losing that knowledge of how the system works is a risk.

> But real world conditions can force code changes Security fixes.

Wasn't last year where a bug in a Java well-used lib (and pretty easy to use) caused mayhem in every SRE/SEC services in every banks? It was a fun two weeks. I worked like 50 hours the first three days.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#526

Earlier quoted context omitted.

Nothing beats an internet random telling us how much a successful person is silly and also can't meet their own superior standards. It's not about being a fan or not, it's that you're not actually providing any real insight other than signalling how smart you are.

These people behave just like the irrational fanboys, except they just do the exact opposite. Being a sheep and being a contrarian sheep are the same thing.

I've no dog in this race but I'm excited to see what influence, if any, this will have on the topology of similarly inflated tech companies.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#527

Now that Elon has cleaned house, when is Twitter hiring again? It might be a good opportunity to get in on the new ground floor.

Who sees the foundation of a high-rise starting to crack and thinks, "perfect time to get in on the ground floor"?

Ground floor is exactly where you don't want to be when the structure collapses under its own weight.

Rather than bring in additional specialists to reinforce it, Musk has partially evacuated the building at least.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#529
> When I joined the team the first project I had was to swap old machines that were being retired for new machines. There were no tools or automation to do this, I was given a spreadsheet with server names.

Writing a program to store a list of servers to be swapped instead of keeping them in a spreadsheet sounds a bit like buying a brewery when you want to drink 1 beer. Program used by a team of one sounds like over-engineering.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#530

Earlier quoted context omitted.

Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes . MBA's will be studying this for years.

Musk might have been right about some things. There probably was some degree of bloat. But to say he's badly mishandled this whole saga is a gross understatement. It is very difficult to utterly kill a site like Twitter; the fact that we're even considering that as a realistic possibility shows just how badly. I think Musk is used to Tesla and SpaceX, which are both companies that a lot of people are (or at least wer…

What is the Mission now at twitter? Kiss Lord Musk's boot?
Post reply on HN