Why Twitter didn’t go down: From a real Twitter SRE

Post by saagarjha » Tue, Nov 22, 2022, 11:26 AM UTC

bombolo wrote: post #677

TranquilMarmot wrote: post #375

Earlier quoted context omitted.

We used to have a horribly written node process that was running in a Mesos cluster (using Marathon). It had a memory leak and would start to fill up memory after about a week of running, depending on what customers were doing and if they were hitting it enough. The solution, rather than investing time in fixing the memory leak, was to add a cron job that would kill/reset the process every three days. This was easier…

USA at some point had an anti-missile system that needed periodical reboots because it was originally designed for short deployments, so the floating point variable for the clock would start to lose precision after a while.

Post by scott_w » Tue, Nov 22, 2022, 11:28 AM UTC

carabiner wrote: post #477

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes . MBA's will be studying this for years.

Post by kranke155 » Tue, Nov 22, 2022, 11:29 AM UTC

rjzzleep wrote: post #429

poulsbohemian wrote: post #87

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wh…

Post by Havoc » Tue, Nov 22, 2022, 11:32 AM UTC

hairofadog wrote: post #105

Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is: https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc... ) If this is the new town square, you can forward my mail to a cabin in the woods.

Post by mrazomor » Tue, Nov 22, 2022, 11:37 AM UTC

mjr00 wrote: post #451

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

It's crazy to think about, but many people who use and build software today, including HN readers/commenters, are young enough to have only been exposed to the SaaS, cloud-first era, where software built with microservices deployed from CI/CD systems multiple times per day is just the way things are done. You're totally right; if you don't make changes to the software, it's unlikely to spontaneously stop working, esp…