Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

221–230 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#221

Post by Operative0198 »

HyperSane wrote: post #192
throwawaylinux wrote: post #109

Earlier quoted context omitted.

I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.

It would be funny for Musk to share privately with friends. It is a deeply strange and inappropriate thing for the CEO of a billion dollar company to post publicly. Something is deeply wrong with Musk's decision making process.

He can get away with posting just about any silly thought because he has no board of directors to answer to and no one can fire him. Big difference when you are thinking of him as a CEO.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#222

Post by HyperSane »

nverno wrote: post #86
ulkesh wrote: post #27

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…

Judging by some of the old patents he's filed [1], I'd guess he has at least a decent understanding of the tech involved. Probably less so, when it comes to the details of more modern distributed systems, but I also wouldn't be surprised if he's spent some effort towards all that as well - he's been working in/around pretty cutting edge tech for quite a while. Could he sit down and code it himself? probably not, but…

Musk was fired from PayPal because he wanted to replace all the Unix servers with Windows.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#223

Post by jdminhbg »

abakker wrote: post #199
jdminhbg wrote: post #152

Earlier quoted context omitted.

> How can us commenters on HackerNews sit from our armchair and say "ah, goofballs should've just not let headcount get so high!" The cliche HN comment on sites like Twitter (and many, many others, any time headcount comes up) has always been "why do they need so many people?" I've mostly dismissed it the same way I dismiss "I could build Uber in a weekend," but with every other tech giant laying people off, maybe I…

I think this is not too far wrong. I also think in addition that the really competitive job market of the last 2 years and the hiring they were doing inevitably resulted in some highly paid people who were not good fits or good at their jobs. I think that covid wage hikes and job competitiveness is doing a bit of reckoning now as there is finally enough data to evaluate performance relative to output. OTOH, at non te…

I'm not doing interviews or surveys, but this also matches with my anecdotal experience with non-"tech" companies, who are all still starving for talent. I guess it makes sense, given that tech companies disproportionately pay in stock and tech companies were disproportionately highly-valued during the 2020-22 Covid weirdness.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#224

Post by afavour »

libraryofbabel wrote: post #178
fenomas wrote: post #139

Earlier quoted context omitted.

To me the analogy here is that the new boss rolled in and sold all the fire extinguishers. That by itself doesn't set the building on fire - it doesn't even increase the chances of a fire occurring on any given day. But when one does...

Every SRE knows that the leading cause of outages by far is someone making a change to the system. Twitter isn’t shipping many new features right now or even doing much maintenance. But eventually they will have to. So the analogy becomes, the new boss sold all the fire extinguishers and also placed a short temporary ban on cooking in the building . But eventually people are going to start turning on stoves again… an…

> Twitter isn’t shipping many new features right now

Is that true? I thought one of Elon’s big pushes was launching the whole Blue Tick subscription thing. That doesn’t feel like a small feature.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#225

Post by grayfaced »

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

But real world conditions can force code changes. For example, a region abandons daylight savings time or a court order on copyright infringement. Someone unqualified working a system they are unfamiliar with could blow it up. Losing that knowledge of how the system works is a risk.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#226

Post by HyperSane »

Operative0198 wrote: post #221
HyperSane wrote: post #192

Earlier quoted context omitted.

It would be funny for Musk to share privately with friends. It is a deeply strange and inappropriate thing for the CEO of a billion dollar company to post publicly. Something is deeply wrong with Musk's decision making process.

He can get away with posting just about any silly thought because he has no board of directors to answer to and no one can fire him. Big difference when you are thinking of him as a CEO.

I understand how he can get a way with it, just not his thought process.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#227

Post by inferiorhuman »

A4ET8a8uTh0 wrote: post #132
emodendroket wrote: post #60

Earlier quoted context omitted.

I'll put my cards on the table and say I would have been just as happy to see him fail as anyone else, but it did begin to take on the tenor of the constant, never-came-true stories about how any minute now they were going to spring the trap on Donald Trump and he'd get thrown out of office. e: I did not mean for this to be an invitation for everyone to argue about the merits or demerits of Donald Trump.

I personally have somewhat mixed feelings. I do enjoy a good dumpster fire like any good internet denizen ( and I won't deny that there is an odd level of.. excitement maybe.. that one major social platform may go down ). I have no love for social media as some of my posts may indicate and for a variety of reasons including the amount of oxygen this creation takes and the amount of confusion it causes in general popu…

I was about to say that you're talking about two different problems. But, no, what's happening to Musk is the same thing that happened to Trump. Their inner circles knew how rotten they both were, but outside of that they were largely unknown.

New Yorkers despise trump and have for ages. From tenants, to contractors, to business partners they all know what a shitheel he is. Outside of NY though trump was largely seen as an affable caricature of a rich dude. Case in point: NYC as a whole isn't averse to republicans and yet they still rejected trump in 2016.

Elon's the same way. Sure he called that diver a pedophile. But he also had a Tesla employee hounded until he fled the country. Musk had the guy swatted for fucks sake (or tried to at least). Or what about the journalist who pissed him off and ended up having some Tesla features disabled on his car. Or his ex-wife who talked about being treated like an employee. These things were mostly inclined to stay in the background until Musk elbowed his way into the spotlight. Now the whole world can see that he went full on Howard Hughes.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#228

Post by StanislavPetrov »

MuffinFlavored wrote: post #131
swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

> Twitter grew 3x on the headcount front There were multiple executives making $10m/yr+ There were board members There were shareholders Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"? These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc. How can us commenters on HackerNews sit from our arm…

>Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"?

For the same reason that colleges and universities have seen their administrative bloat skyrocket at 10x the rate of student enrollment. Administrative bloat inevitably creeps into all large organizations. Many of the people in the trenches making hiring decisions weren't considering the overall financial performance of Twitter as a company. They were making hiring decisions based on what was happening in their own department, or how that decision would help advance their own agenda, or increase their budget, or increase manpower on a favored project. When you further consider that many at Twitter openly conceded (and in many cases, bragged about) that they viewed their role at Twitter as moral arbiters of society, crucial to policing the discourse of the public, it is not hard to see how enlisting as many true believers as possible to the cause would be seen as desirable, regardless of the larger financial implications.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#229

Post by kyriakos »

sroussey wrote: post #101
porknubbins wrote: post #69

Earlier quoted context omitted.

The default, naive assumption should always have been programs keep running indefinitey on their own. If thats not the goal of software then I don’t know what is (might as well go back to switchboard operators). Real world experience tells us that, to the contrary, all software goes down and requires specialist intervention eventually. I think a lot of people just jumped to the second level based on political motivat…

I had MySQL running on some bare metal for many years without a restart. I was terrified to update the kernel at that point, knowing that system disk had been running continuously for many years, and had no faith it would restart successfully. Finally got two new servers to replace these (with these new SSD things!) and after migration, sure enough, one of the old servers failed to boot.

Even if your mysql instance and hardware had run indefinitely, if a table is being written to it will eventually run out of disk space or key space and crash. How long it will take depends on the application but it will happen eventually and if no one is around to fix it...

Re: Why Twitter didn’t go down: From a real Twitter SRE

#230

Post by ggm »

The article is good, and informative but a little odd in one respect. It uses air-quotes to introduce concepts like "rack" and "shelf" in a DC, but does O(logn) notation mid-flow.

If you don't know what a rack is, how are you meant to know what the Order of scaling function means? Thats a highly computer sciences specific notation, and if you grok O(n) you know what a rack, a host, a DC is.

Post reply on HN