Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

661–670 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#661

Post by pas »

Lutger wrote: post #611
H8crilA wrote: post #573

Earlier quoted context omitted.

This is a PR article that tries to push the idea that Twitter is OK (to work at, and maybe also to buy ad space from). Damage control.

I read it as the exact opposite: the reason Twitter is still ok* is not because all these people were just browsing reddit at work. You can't just gut Twitter to run on only a couple of hundred people and still expect the same results in the longterm. Twitter was not a leftist commie welfare company as Musk and its fans want it to be. It was actually the fine work SRE (amongst others) put into it that makes it still…

It could have been both! :)

And seriously, the gaming industry is infamous for its "year round crunch mode". It can be done. Many sectors are "hard work" (ie. public sector, fundamentally non-tech gigacorps with no real in-house IT competence, etc.)

Twitter was an innovator. I remember getting into Scala party because of some open source code library from them back then, I don't know, probably around Scala 2.8.

That was absolutely unnecessary for their core business. All of this can be done in bad C++, or plain old Java 7, or whatever, for low cost, built in Bangalore.

Now that would of course likely not withstand this kind of change storm that Elon caused. But that's a different story. (Also what Elon probably doesn't understand on a technical level (but likely guessed from a business aspect), is how flexible Twitter is. In this regard he hit arrived at the right time. Twitter is fundamentally the same as it was, now it's time to innovate with it, and the architecture he found is exactly the right one for that.)

Re: Why Twitter didn’t go down: From a real Twitter SRE

#662

Post by [deleted] »

userbinator wrote: post #19
danielodievich wrote: post #3

Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.

software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…

[deleted]

Re: Why Twitter didn’t go down: From a real Twitter SRE

#663

Post by Yoric »

holoduke wrote: post #506
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

Not tech but I'm convinced that just answering LEO requests from all over the world takes at least 500 people. That's without their managers, payrolls, etc.

Likewise, bringing in Ad money would be a few more hundreds, because you need to chase leads in all countries.

Getting the Ads to work? That's tech and I'd be surprised if it was less than 100 people, too.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#664

Post by TheOtherHobbes »

tempestn wrote: post #452
28304283409234 wrote: post #419

Earlier quoted context omitted.

Argh. "It works now, so it will work until forever." It takes _effort_ to make it work this smoothly now, _and in the future_. SRE is about _preventing_ issues. Not mopping up after them. To me, the article read like every succesfull sysadmin story: there's no fires, so sysadmin must be bloat.

I think you're misinterpreting the comment you're replying to. They would agree with you that the tiny SRE team described in the article sounds very effective, and likely have a lot to do with why the site is still up and running currently. Work like that should continue. But if 1-3 people can have that degree of impact, what are the other 8000 doing? (Again, this is just me attempting to interpret the point made by…

It's not 1-3 people. The entire SRE team globally - including the technicians and the engineers with server access - is easily going to be in the hundreds.

The SRE manager is in charge of keeping it all running. He isn't running around the world swapping out servers. He also isn't sitting back with his feet up thinking "All done - now how are my Pokemon doing?"

It's a dynamic process with quality monitoring, budgeting and reports, post-mortems, continual experiments to see if uptime can be improved, and redesigns as hardware and software change.

It's part of the backend, but is only loosely coupled to the content management and delivery system, the ad machine, moderation, marketing, and so on, all of which are going to have similarly complex structures.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#665

Post by throwuwu »

watt wrote: post #644
_3u10 wrote: post #593

Earlier quoted context omitted.

If you consider that engineers cost about $1000 a day and he got rid of 4,000 of them… oddly it works out to about $4 million per day. I’m not an expert in math but it’s seems pretty possible. Plus they’ll have a lot more impressions to sell now that people are allowed to speak. Ad rates might drop but then someone will write an article about how they are getting great CPC on Twitter and everything will be back to no…

Dead cat bounce. Engineers are paid $1000 a day because they create value to the tune of $10 000 a day. So maybe you saved $4 million a day in very short term, but now the momentum is gone, and soon you will not be earning $40 million a day from the value that now was not being created, just in a "few" short months. I wish Elon the best, but he could have hired his own team of "hardcore" engineers and put them to wor…

SWE are not assembly line workers, they are the people that build the automated assembly line. It’ll keep functioning well enough without them and with a much smaller team performing maintenance. I’d even bet on new feature being released faster now that they’ll be owned by highly motivated individuals instead of committees of managers and randomly assigned engs.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#666

Post by threeseed »

holoduke wrote: post #506
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

> I am convinced you can run the entire tech stack with a team of a 100 people

Please tell us in detail about the Twitter stack.

Because I always find it fascinating how people think they can estimate the effort to maintain it whilst having next to no understanding what so ever of the tech stack.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#668

Post by mazurnification »

jillesvangurp wrote: post #514
ulkesh wrote: post #27

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…

You underestimate Elon Musk. Many people have done that before and lost that bet. If anything, he repeatedly succeeded in building world class software and hardware teams for Tesla, SpaceX, and a few other companies. The notion that he won't be able to attract world class talent is ludicrous. Yes, he is a bit of a liability and his management style is obnoxious and unconventional. But he does get things right once in…

Agree, I wonder how much it is thought through strategy and how much is just "natural" style applied indiscriminately. I think one more important part is that he has money/resources to be able to make mistakes without bankrupting and stubbornness to plough thru even when things go wrong.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#669

Post by nebula8804 »

mirzap wrote: post #402
poulsbohemian wrote: post #87

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

Quoted post unavailable.

I don't know man. His last pure software company was PayPal and they fired his ass from the CEO role and put in a competent CEO (Peter Theil) to right the ship and sell it off before the 01 crash.

Tesla and SpaceX have been wildly successful and he will forever recieve some well deserved credit for their successes but at the same time I think its pretty bad that Tesla (and SpaceX) have had multiple near death experiences and some them could have been avoided if it weren't for incompetence.

I followed the Model 3 production religiously and the amount of dumb mistakes they made even after other industry experts were yelling at them about imminent iceberg collision is just unbelievable. In fact the Model 3 production, with its horribly designed body that he wasn't keeping on top of and Musk's stupid attempts to automate all of assembly should be entirely attributed to him. It was so bad it was worse than even the shortseller's worst predictions.

I guess you can give him a pass for near bankruptcy #1(Roadster roll out) and #4(COVID lockdowns). Not sure about #2(Model S early sales woes and potential acquisition by Google)

All I am saying is that he fucks up A LOT and has thus far managed to stave off the consequences of his decisions but many occurrences have been down to luck and luck is something you should never bet on.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#670

Post by bandrami »

drcross wrote: post #592
Moissanite wrote: post #580

Earlier quoted context omitted.

The fact that he was able to "buy" Twitter and yet transfer a significant amount of debt to the company rather than being liable himself is just another sign of how different "rich people accounting" is. He will walk away from this with a bit less theoretical money but no material impact to his life, while thousands of people are having their lives up-ended. How long are we going to keep letting shit like this happen…

Debts from where? Any evidence of this at all? It seems like fantasy considering how much money the rest of Musks companies have in the bank.

LOL no. Rich people don't have money, they own companies. In fact they go to nearly absurd lengths to avoid having money as much as possible.
Post reply on HN