Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

631–640 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#631

Post by headsoup »

threeseed wrote: post #423
memish wrote: post #284

Earlier quoted context omitted.

It's actually scary how many people, even engineers, put their reputation on the line saying Twitter wouldn't survive the weekend. It wasn't just Twitter employees. It's like a mass psychosis of some kind. It comes off as a kind of desperation, as though they need Elon to fail. Why? What's driving that response?

> Why? What's driving that response? Experience. Many of us have worked at companies where there is a lot of duct tape holding things together and when you let go of entire teams (not just a large percentage) then it isn't unreasonable to be pessimistic. Especially when you know that in order to fix problem A you need to take B, D, E, C corrective actions in that order. And you learnt that through years of things goi…

Just curious, among the critical stuff just holding on, was there also a whole pile of departments and teams doing work that could cease tomorrow and the company would blink and move on?

My experience is both exist at the same time because the leadership teams don't actually know what core business is or are busy building empires and resumes.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#632

Post by Lutger »

mjr00 wrote: post #451
throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

It's crazy to think about, but many people who use and build software today, including HN readers/commenters, are young enough to have only been exposed to the SaaS, cloud-first era, where software built with microservices deployed from CI/CD systems multiple times per day is just the way things are done. You're totally right; if you don't make changes to the software, it's unlikely to spontaneously stop working, esp…

Absolutely. I remember we build a unimaginably brittle application many years ago, I think it was running on Windows XP and glued together a complex system with COM calls into this single page webapp even before react was a thing. It was build on a very small budget, serving the core business of a very tiny company.

Like maybe 8 years later I found out it was still humming along happily, without really even a sysadmin attending to it, on a single workstation using consumer hardware, servicing the company that had grown tenfold in size.

It blew my mind it still just worked all these years.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#633

Post by Xylakant »

orwin wrote: post #500
fnord123 wrote: post #472

Earlier quoted context omitted.

Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?

I worked for a bank with 28000 total engineers. I know because part of what i did was crunching github and gitlab data and detecting double accounts (and deleting account from people not there anymore). I'd say 3/4th of them were just cost of doing business. Including me (my role wasn't really necessary, the stuff i did was cool and everything, but a bit useless).

At this scale, having someone take care of deleting double/expired accounts is just good hygiene and I would not consider that useless. Forgotten accounts are a security risk.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#634

Post by chrsig »

andreyk wrote: post #577
Dave3of5 wrote: post #571

I don't think it will fail on a technical level. As this article says lots of engineering has gone in to make the thing pretty resilient. I would also say that there are still enough engineers who work there who can figure out what's gone wrong and "turn it on and off again" or w/e makes it splutter back to life. In terms of changes to the Platform ditto. It's not difficult to make these changes that a team of 100's…

Yep, agree. Twitter's revenue was primarily from ads, and now I'd bet their ads revenue has dropped a huge amount. Given how Musk is behaving now (reinstating Trump etc.), and losing many pivotal sales people, and the firing of a ton of people in charge of dealing with hate speech and such, it seems unlikely advertisers will return now.

advertisers wont care in a couple months. the situation has lots of public attention now, which is what the advertisers are actually afraid of. They don't want their brand associated with the craziness. They don't actually care one way or the other about any choice musk makes. They're not going to just walk away from a 300+ million person audience permanently just due to principles

Re: Why Twitter didn’t go down: From a real Twitter SRE

#635

Post by duckmysick »

userbinator wrote: post #19
danielodievich wrote: post #3

Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.

software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…

> It's possible to make things which last.

Things last when you take good care of them.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#636

Post by wikfwikf »

_3u10 wrote: post #587
wikfwikf wrote: post #553

Earlier quoted context omitted.

data architecture is bloat? Have you implemented a system which stores hundreds of billions of pieces of media content and makes different slices of them immediately available to hundreds of millions of users?

No but the people who made clickhouse did. I don’t think 1000s of engineers were required for that. Where’s the 1000s of engineers for Postgres? Most stuff that works is made by a handful of people. Look at io_uring it’s basically one guy at Facebook…

There are tens of thousands of engineers maintaining and supporting postgres and clickhouse in all the organizations where they are used, the vast majority of which are not on Twitter's scale.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#637

Post by hackerlight »

hnbad wrote: post #621
_3u10 wrote: post #601

Earlier quoted context omitted.

I don’t think given their ad sales that the sales people were that pivotal.

As I understand it, most of the ad slots were sold at a yearly sales event. This time most advertisers were cautious because the company couldn't respond to questions of how Elon Musk's takeover would affect the company (and specifically the type of content alongside which the ads would appear) as this was happening before he took over but after the sale was practically a done deal. This left them with only a fractio…

  "Additionally it seems a lot of advertisers have cautioned their ad departments to halt spending"
What percentage? Big difference between 0.03 and 0.3.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#638

Post by christophilus »

mgkimsal wrote: post #589
mjr00 wrote: post #451

Earlier quoted context omitted.

It's crazy to think about, but many people who use and build software today, including HN readers/commenters, are young enough to have only been exposed to the SaaS, cloud-first era, where software built with microservices deployed from CI/CD systems multiple times per day is just the way things are done. You're totally right; if you don't make changes to the software, it's unlikely to spontaneously stop working, esp…

The other side of that is browsers. Even if you don’t change your code, the platform people are running your code in changes, automatically in many cases. New JS or CSS behavior in next safari or chrome? You need to patch/push to accommodate running environments that are outside your control.

The old space jam site worked for ages, and would still work if they hadn’t taken it down. The web is pretty good about keeping backwards compatibility.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#639

Post by bigiain »

_3u10 wrote: post #587
wikfwikf wrote: post #553

Earlier quoted context omitted.

data architecture is bloat? Have you implemented a system which stores hundreds of billions of pieces of media content and makes different slices of them immediately available to hundreds of millions of users?

No but the people who made clickhouse did. I don’t think 1000s of engineers were required for that. Where’s the 1000s of engineers for Postgres? Most stuff that works is made by a handful of people. Look at io_uring it’s basically one guy at Facebook…

The Postgres engineers are not responsible for running their code on a shared platform supporting a billion users.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#640

Post by TeMPOraL »

hnbad wrote: post #604
Moissanite wrote: post #580

Earlier quoted context omitted.

The fact that he was able to "buy" Twitter and yet transfer a significant amount of debt to the company rather than being liable himself is just another sign of how different "rich people accounting" is. He will walk away from this with a bit less theoretical money but no material impact to his life, while thousands of people are having their lives up-ended. How long are we going to keep letting shit like this happen…

Leveraged buyouts are certainly a thing that sounds like it shouldn't be possible the first time you hear of it. It seems extremely odd that you can buy a company with money you don't have and then have the money take on that debt instead of having to take it on yourself. As I understand it, this works by rounding up potential loans, approaching the board of the company and getting them to sign over ownership of the…

> Intuitively this shouldn't be possible as it's acting against the company's own self-interest despite being in the interest of the shareholders (and the buyer), but I think "the company's interest" in practice is defined by "the owners' interest" (and the owners in this case are the shareholders, who sell the company). I guess corporations aren't people after all.

Or maybe they are too much like people. Right now there's a bunch of things that I should be doing, that would be in my best interest - continue with my TODO list, or do some exercises. Instead, I'm browsing HN. This shouldn't be possible, but it is, because I'm a human - what I want to do, what's in my best interest, and what I actually do are three different things, and rarely aligned.

(Ironically, in humans this is usually called an issue with executive functioning, whereas in companies, it's the reverse.)

Post reply on HN