Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

811–820 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#811

Post by shagymoe »

When I was learning programming, I "wrote" an internal app where a supervisor input some information in a form, it connected to an oracle database and then printed out an outbound manifest for a truck full of car parts.

I put "wrote" in quotes because what I had actually done was install Apache and PHP Nuke on a Windows NT server and then modified an existing form page to do what I wanted.

I wrote that application in 2002 and never had a problem with it. I never restarted Apache or the server or did any maintenance. I didn't even upgrade Windows NT if I'm being honest. Windows NT became unsupported in something like 2006 and I left the company in 2010.

I received a call in 2018 from the last person still working there who knew who I was. It had finally fallen over and they wanted to know if I could help. That was the first time it ever had an issue.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#812

Post by lcw »

Varriount wrote: post #386
lcw wrote: post #366

Earlier quoted context omitted.

Agreed, one of the craziest bugs I had to deal with was we had a distributed system using lots of infrastructure. Said distributed system started having trouble communicating with random nodes and sub-systems. I spent 3 hard days finding a Linux kernel bug where the ARP cache was not removing least recently accessed network addresses. Normally, this wouldn't be a big deal for a typical network because few networks wo…

Goodness, what kind of process/tools did you use to track that problem down?

We didn't have much tooling outside of typical things you would find in a Linux distro. It started with trying to isolate a node having issues. Then looking at at application and kernel logs. Then testing the connection to the other node via ping or telnetting a port I knew should be open. Found out I couldn't route then just process of elimination from that point till we managed our way to looking at a full ARP cache. Tested that we could increase the ARP cache size to fix the issue. Then figured out by going through the kernel why it wasn't releasing correctly by looking at the source code for the release we were using. I'm simplifying some discovery, but there was no magic unfortunately.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#813

Post by vidarh »

joenot443 wrote: post #763
matwood wrote: post #752

Earlier quoted context omitted.

I agree those were odd takes. I've likened firing most of the engineers to taking your hands off the wheel in the car. It won't crash immediately, but it doesn't mean the car can go driverless. With that said, there are differences between internal systems and something like Twitter on the public internet. I assume that Twitter is a system under constant attack. What happens when the next log4shell level vulnerabilit…

If Twitter went another month without an outage, how would you adjust your opinion? How about a year? The car analogy is amusing, but how much does it really hold up? Have we ever seen another major social media company drop this much of its staff in one go? I certainly can’t think of an example. I think we’re in somewhat uncharted waters here. A driverless car won’t last long, we know that for a fact. I think it rem…

> If Twitter went another month without an outage, how would you adjust your opinion? How about a year?

It's a tricky one, because on one hand it increase my trust that their system was built robustly, but at the same time the passage of time would increase the chance of unseen/unaddressed "wear an tear" (bot figurative and literal) that might be going unaddressed, or under-addressed. But we have no view into that.

We won't really know until they suffer a major problem whether or not they have enough staff yet to keep sufficient maintenance going that such an event doesn't cascade into something much worse and/or whether or not they will be able to recover from it in a reasonable amount of time.

Horrible systems can survive, but often they survive through sheer luck.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#815

Post by w0m »

resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

If you only care about 'it mostly didn't crash' as the end-goal of a company, that would be a reasonable take.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#816

Post by blueprint »

donkeyd wrote: post #372
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

if 80% of knowledge is missing due to 80% of people being gone then the team who built it failed to document or automate themselves out of a job, meaning even other people on the team in the good ol' days would still have faced the same issues and merely had to chase down the original authors. that doesn't sound like it would be completely true. maybe 5-20% of un-documented knowledge walked out the door? completely rough guess. but even 0.5% of knowledge can sometimes be critical.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#817

Post by bena »

coldtea wrote: post #535
oxfordmale wrote: post #443

Earlier quoted context omitted.

Let's do a thought experiment and see what functions aren't needed to keep the light on for 30 days: 1) HR 2) Legal 3) Sales 4) Marketing 5) Payroll 6) Admin staff 7) Most of Engineering, other than the bare minimum of L1/L2/3 support. As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend.

> As someone paraphrased, a car without breaks and steering wheel works just fine until you hit the first bend. On the other hand, a car without a second and third steering wheel, 20 windscreen wipers, and an oven in the back, keeps running just fine, even after the first bend...

But the trick here is to discover which analogy we're dealing with.

We don't know. It could be a situation with one steering wheel and 40 wipers. And as long as it's the wipers that fail, we're fine.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#818

Post by dopidopHN »

Aeolun wrote: post #719
itsoktocry wrote: post #676

Earlier quoted context omitted.

> removing engineers won't instantly crash the product. It'll happen slowly It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.

> thought that Twitter would crash within days of the engineers being fired Because they work for companies where the product would fail within days of them being fired themselves.

I’ve been in a lot of sinking ship.

For the type of jobs at hand here, One of the things I learned is that nobody is essential. Even that person you think is essential.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#819

Post by vonmoltke »

devoutsalsa wrote: post #7
cloudking wrote: post #4

Earlier quoted context omitted.

Nice work! Does Twitter runs their own DCs or hosted somewhere else?

As of three years ago, Twitter had 2 to 3 data centers, and was moving some stuff to GCP. Not sure if current state of things.

There are currently three data centers, hosting all of the real-time and most of the batch production load. Ad-hoc load and some production batch load was migrating to GCP, but this was being significantly curtailed in the months before the acquisition closed because it turned out to be very expensive, much more so than anticipated and more than the equivalent workloads had cost in the data centers.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#820

Post by mlindner »

akira2501 wrote: post #476
minhazm wrote: post #373

Earlier quoted context omitted.

Many people who have worked with Musk have shared similar sentiments in interviews. But it seems that people just refuse to believe any of it. People think that there's no way it's possible for someone to be that deeply technical and be a CEO of multiple companies at the same time. I've talked to people about it and they straight up refuse to believe it saying that it's impossible and that any evidence of him being t…

With a handful of tricks or a patsy in your pocket, it's easy enough to pull things like this off. These are all self-reported encounters, which lends some doubt to them; as I've never seen any public performance of his that suggests he has this exceptional intelligence or that he isn't subject to the same amount of irrational thinking that most humans are. You may be able to do some type of rocket equation in your h…

Maybe I'm a particularly dull engineer, but I've taken several aspects of personal advice from what he has said in interviews (the especially technical ones, not the ones aimed at a mass audience where he repeats his standard canned speech) and found them useful for myself personally.

Here's two examples I've found particularly insightful that shows he has some ability to talk about engineering details.

This example where he talks about the choice of steel for Starship as opposed to any other metal, something that would be an otherwise unsual choice: https://youtu.be/vLC5W53Fsyg?t=936

This example that I've personally incorporated into my own thinking where he talks about his "five step process" for engineering design refinement (watch at least until he starts talking about Tesla Model 3 battery stuff): https://youtu.be/t705r8ICkRw?t=805

Post reply on HN