Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

431–440 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#431

Post by broodbucket »

colesantiago wrote: post #117
throwaway0x7E6 wrote: post #82

Earlier quoted context omitted.

why? a single app running on a single server is several orders of magnitude more resilient than a spaghetti clusterfuck of services upon services. twitter could be brought down by a single expired certificate

most of the mastodon instances I saw were: - ghost towns with little to no one there. - didn't open registrations. - completely dead with ssl invalid certificates with expired domains. So you would have to keep moving to another mastodon instance (if you're lucky) or try and run your own instance and join the many instances with the three issues above. There is no monetary incentive to keep a mastodon instance runnin…

There's a gigantic flood of new users (i.e. literally multiplying the userbase) in the past few weeks, so yeah, a lot of servers are restricting signups to cope.

I don't think "begging for donations" needs to scale, if instances get too large to keep running then smaller instances should (and do) split off, they can still talk to each other after all.

There's an actively maintained public instance list here: https://joinmastodon.org/servers

Re: Why Twitter didn’t go down: From a real Twitter SRE

#432

Post by ithkuil »

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

An example where something that correlates with time can reveal pre-existing bugs long after the system was chugging along just fine: counter limits/overflows.

Simple example: you have a DB with a table with an auto incrementing table. You chose a small integer type for the primary key and after years this just worked fine, you finally saturate that integer type you can no longer insert rows in the table. Imagine now this has cascading effects in other systems that depend on this database indirectly and you end up with an "outage"

Re: Why Twitter didn’t go down: From a real Twitter SRE

#433

Post by gitgud »

mirzap wrote: post #402
poulsbohemian wrote: post #87

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…

Quoted post unavailable.

No post body was provided.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#434

Post by ShredKazoo »

abakker wrote: post #199
jdminhbg wrote: post #152

Earlier quoted context omitted.

> How can us commenters on HackerNews sit from our armchair and say "ah, goofballs should've just not let headcount get so high!" The cliche HN comment on sites like Twitter (and many, many others, any time headcount comes up) has always been "why do they need so many people?" I've mostly dismissed it the same way I dismiss "I could build Uber in a weekend," but with every other tech giant laying people off, maybe I…

I think this is not too far wrong. I also think in addition that the really competitive job market of the last 2 years and the hiring they were doing inevitably resulted in some highly paid people who were not good fits or good at their jobs. I think that covid wage hikes and job competitiveness is doing a bit of reckoning now as there is finally enough data to evaluate performance relative to output. OTOH, at non te…

>OTOH, at non tech companies, all the research interviews and surveys I have done recently continue to show a talent shortage and fears of losing tech staff

I wonder if any science fiction writer could have predicted silliness like Snapchat having fewer outages than serious government websites because Snapchat is better at hiring engineers. Tech hiring is so bizarre.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#435

Post by threeseed »

intelVISA wrote: post #426
kibibyte wrote: post #400

Earlier quoted context omitted.

I can imagine both cases being true, that the renewal process is automated and that certs won't get renewed because institutional knowledge has left the door. Where I'm at, service-to-service TLS certificates (the bulk of our certs) are automatically rotated by our deploy systems. But there are always the edge cases: the certificates manually created a long time ago (predating any standardized monitoring systems) wit…

> certs won't get renewed because institutional knowledge has left the door. The parallel reality where you need to be a veteran SRE with an MIT degree to operate the arcane tool 'certbot'.

They aren't talking about the front-end certificates which expire in Feb 2023.

It's likely the ones to encrypt all of the traffic involving the Finagle micro-services, data sources, observability systems etc. And I suspect the issue there is that you are going to need to do a rolling restart.

Which I personally would not want to be doing if 90% of the company is no longer there.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#436

Post by j3th9n »

aeyes wrote: post #404
TranquilMarmot wrote: post #375

Earlier quoted context omitted.

We used to have a horribly written node process that was running in a Mesos cluster (using Marathon). It had a memory leak and would start to fill up memory after about a week of running, depending on what customers were doing and if they were hitting it enough. The solution, rather than investing time in fixing the memory leak, was to add a cron job that would kill/reset the process every three days. This was easier…

You don't even need that, the kernel OOM killer would take care of this eventually. Unless its something like Java where the garbage collector would begin to burn CPU.

The OOM killer doesn't restart (randomly, unless configured) killed processes, it just kills.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#437

Post by mhio »

swellguy wrote: post #34

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

Have a browse through their engineering blog: https://blog.twitter.com/engineering/en_us

It's largely focussed on the event stream behind the core service and data analytics. There's maybe one entry on the main data store and one on search over the last few years.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#438

Post by threeseed »

grayfaced wrote: post #225
throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

But real world conditions can force code changes. For example, a region abandons daylight savings time or a court order on copyright infringement. Someone unqualified working a system they are unfamiliar with could blow it up. Losing that knowledge of how the system works is a risk.

> But real world conditions can force code changes

Security fixes.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#440

Post by resonious »

donkeyd wrote: post #372
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

> how does it persuade you of that?

By flowing from many people think it's bloat - I'll tell you what's really going on to tiny team of 1~3 built whole infra for critical component.

I'm not really trying to make commentary on whether or not Twitter engineering was bloat, or whether or not I think it'll hit problems in the future. Just commenting on the fact that the article broke my expectations a little bit as a reader.

Post reply on HN