Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

991–1000 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#991

Post by hackfrednews »

memish wrote: post #326
ulkesh wrote: post #27

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…

John Carmack, "Elon is definitely an engineer. He is deeply involved with technical decisions at spacex and Tesla. He doesn’t write code or do CAD today, but he is perfectly capable of doing so." Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon i…

Channing Robertson, the face of Stanford chemical engineering department and the associate dean of Stanford’s School of Engineering, who taught and mentored Elizabeth Holmes, has said the following to say about her:

“She had somehow been able to take and synthesize these pieces of science and engineering and technology in ways that I had never thought of.”

“I never encountered a student like this before of the then thousands of students that I had talked”

“You start to realize you are looking in the eyes of another Bill Gates, or Steve Jobs.”

He also maintained that Holmes was a once-in-a-generation genius, comparing her to Newton, Einstein, Mozart, and Leonardo da Vinci.

Excerpt from: "Bad Blood: Secrets and Lies in a Silicon Valley Startup" by John Carreyrou.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#992

Post by dotancohen »

infotropy wrote: post #906
skeeter2020 wrote: post #898

Earlier quoted context omitted.

I know the bus factor is the morbid "how many people can get hit by a bus?" idea, but I actually like to present it as "how wide is the bus?" in terms of people being the conduits along which information and instructions flow; a wider bus provides redundancy. Aside from being more positive I think it's also more accurate for how we want teams to work. The "win the lottery and quit" metaphor is just stupid.

I usually rephrase it as “getting run over by the lottery” or “winning a bus”. Just to be different/humorous/less morbid.

Phrasing "getting run over by the lottery" sounds like a good humorous, compromise. But "winning a bus" sounds like you're actively rooting for the person to be hit. I'd stick with the former. ))

Re: Why Twitter didn’t go down: From a real Twitter SRE

#993

Post by enominezerum »

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

Takes me back to the first broken mess of an environment I worked in. Change freezes were a day of life and lasted and, magically, nothing would break during that time.

Now, those change freezes even extended to preventative maintenance, one of the dual PSUs in a core switch went bad and we couldn't get an exception to replace it... for 6 months. We got an exception when the second one went down and we had to move a few connections to its still alive mate.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#994

Post by rlt »

polynomial wrote: post #874
donkeyd wrote: post #830

Earlier quoted context omitted.

Good point. I know this as the Mack Truck Theory. For the project I'm working on right now, there's a couple of incredibly valuable people that would cause a pretty significant issue if they disappeared.

Sensitive workplaces (which seems like most these days) have taken to calling this the Lottery Factor (as in team members quit bc they hit the lottery) to spare more delicate types the pain of imagining their peers run over in traffic accidents.

The real world isn’t a safe space, we don’t need to modify our language like this to avoid triggering momentary negative thoughts.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#995

Post by hot_gril »

blisterpeanuts wrote: post #860
justahuman74 wrote: post #392

Earlier quoted context omitted.

> cut corners on actually verifying the subscribers Did they actually verify anything other than the ability to pay $8? It seems wild to me that they thought it would work out just fine

Feels like one of those "ideas that sounded good in the conference room" that turns out to have unintended consequences. Hopefully they can fix this and have an effective way to verify identity... or else just give it up and go back to the old, creaky, somewhat effective method since people are used to it.

It sounds like an ok idea that someone didn't execute properly, unless the idea really was to give the checkmark without real validation.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#996

Post by mmcnl »

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

Sorry for hijacking your expertise, but why no mention of memory leaks? In my experience they can cause really weird bugs not obvious at first, and are difficult to reproduce, i.e. triggered by edge cases that happen infrequently. Or are you assuming services automatically restart when memory is depleted?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#997

Post by throwaway892238 »

mmcnl wrote: post #996
throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

Sorry for hijacking your expertise, but why no mention of memory leaks? In my experience they can cause really weird bugs not obvious at first, and are difficult to reproduce, i.e. triggered by edge cases that happen infrequently. Or are you assuming services automatically restart when memory is depleted?

It depends how well the service was "operationalized":

1) Best case: Monitoring of the service checks for service degradation outside of a sliding window. In this case, more than X percent of responses are not 2xx or 3xx. After a given time period (say, 30 minutes of this) the service can be restarted automatically. This allows you to auto-heal the service for any given "degradation" coming from that service itself. (This does not detect upstream degradation, of course, so everything upstream needs its own monitoring and autohealing, which is difficult to figure out, because it might be specific to this one service. The development/product team needs to put more thought into this in order to properly detect it, or use something like chaos engineering to see the problem and design a solution)

2) If you have a health check on the service (that actually queries the service, not just hits a static /healthcheck endpoint that always returns 200 OK), and a memory leak has caused the service to stop responding (but not die), the failed health check can trigger an automatic service restart.

3) The memory leak makes the process run out of memory and die, and the service is automatically restarted.

4) Ghetto engineering: Restart the service every few days or N requests. This extremely dumb method works very well, until you get so much traffic that it starts dying well before the restart, and you notice that your service just happens to go down on regular intervals for no reason.

5) The failed health check (if it exists) is not set up to trigger a restart, so when the service stops responding due to memory leak (but doesn't exit) the service just sits there broken.

6) Worst case: Nothing is configured to restart the service at all, so it just sits there broken.

If you do the best practice and put dynamic monitoring, a health check, and automatic restart in place, the service will self-heal in the face of memory leaks.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#998

Post by underwater »

jiggawatts wrote: post #518
underwater wrote: post #497

Earlier quoted context omitted.

Look at his claim that Twitter is slow in India because of poorly batched backend calls [1]. He's simply regurgitating what he thinks he heard from others without really understanding how anything works. He knows less about software architecture than an intern picked at random. 1. https://mobile.twitter.com/elonmusk/status/15921762028730859...

Stop repeating this, because Elon had the gist of it correct, if not the specifics. a) Twitter is atrociously slow for something that displays a mere kilobyte of text at a time. b) Much of that is because of hundreds of client-server JavaScript requests on first load. Not technically RPCs, but just as slow and looking nearly identical on the network. c) Most of the rest is because of a thousand microservices chatting…

He was right on the details of the two different issues, because those were explained to him.

But he conflated two things that he didn't really understand and reached a completely wrong conclusion. He's shared that conclusion with confidence and never admitted he was wrong. That's a fatal flaw for someone who holds a Chief Engineer title at a rocket company.

He's a CEO who is has been incredibly successful at selling a vision and getting people to do what he wants. He is not an engineer. He should stop role playing as one.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#999

Post by philistine »

NaturalPhallacy wrote: post #280
philistine wrote: post #33

Earlier quoted context omitted.

And Twitter's problem are nowhere near technological. The site needed to make more money, not reengineer the whole thing while advertisers are fleeing because Trump is back on on a whim!

>while advertisers are fleeing because Trump is back on [citation needed] CNN's ratings were never better than under Trump. He's fantastic for advertising. So is Musk. All controversial figures are. That, oddly, isn't controversial in advertising. > on a whim! He created a public poll, and when people voted for Trump to be allowed back won, he unbanned him, tweeting "vox populi, vox dei" ("the will of the people is t…

Let me ask you this: did he need to make a poll about Trump's return?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#1000

Post by efreak »

christophilus wrote: post #638
mgkimsal wrote: post #589

Earlier quoted context omitted.

The other side of that is browsers. Even if you don’t change your code, the platform people are running your code in changes, automatically in many cases. New JS or CSS behavior in next safari or chrome? You need to patch/push to accommodate running environments that are outside your control.

The old space jam site worked for ages, and would still work if they hadn’t taken it down. The web is pretty good about keeping backwards compatibility.

Except the original hamster dance, which doesn't display correctly
Post reply on HN