Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

801–810 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#801

Post by KoftaBob »

holoduke wrote: post #506
donkeyd wrote: post #372

Earlier quoted context omitted.

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

How difficult can a platform like Twitter be? I am convinced you can run the entire tech stack with a team of a 100 people.

For comparison, Mastodon is built/maintained by 732 contributors: https://github.com/mastodon/mastodon/graphs/contributors

Re: Why Twitter didn’t go down: From a real Twitter SRE

#803

Post by dekhn »

gfv wrote: post #422
lcw wrote: post #366

Earlier quoted context omitted.

Agreed, one of the craziest bugs I had to deal with was we had a distributed system using lots of infrastructure. Said distributed system started having trouble communicating with random nodes and sub-systems. I spent 3 hard days finding a Linux kernel bug where the ARP cache was not removing least recently accessed network addresses. Normally, this wouldn't be a big deal for a typical network because few networks wo…

Yeah, network scaling bugs are the most fun. The one I liked the most was when after expanding a pool of servers, they started to lose connectivity for a few minutes and then come back a minute or so later as if nothing happened. Turns out we accidentally stretched one server VLAN too wide, to roughly 600 devices within one VLAN within one switch. The servers had more-or-less all-to-all traffic, and that was enough t…

vlans are convenience that is the enemy of performance and undertandability.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#804

Post by nemo44x »

koolba wrote: post #52
nemo44x wrote: post #29

Earlier quoted context omitted.

Turning it off and then back on again probably fixes the issue. There’s very unlikely a grand ticking time bomb just waiting to bring it all down. Recycling servers will probably keep it running. Things do still need to be fixed of course.

> Turning it off and then back on again probably fixes the issue. Turning a large scale system entirely off and on is never simple. Invariably you’ll run into some kind of circular dependency that must be manually investigated. And even tracking those down becomes tricky. Classic examples are things like DNS, service locators, or authentication systems. And large tech companies are notorious for NIH-syndrome for all…

There’s so much redundancy built into modern distributed systems that you can reliably bounce a VM without issue. You can reliably roll bounce a series of VMs.

Twitter doesn’t have unique scale problems by todays standards.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#805

Post by loudmax »

matwood wrote: post #752
itsoktocry wrote: post #676

Earlier quoted context omitted.

> removing engineers won't instantly crash the product. It'll happen slowly It's amazing to me how many people following the Twitter saga, some familiar with or actually working in technology, thought that Twitter would crash within days of the engineers being fired. And because it didn't, the job cuts are justified.

I agree those were odd takes. I've likened firing most of the engineers to taking your hands off the wheel in the car. It won't crash immediately, but it doesn't mean the car can go driverless. With that said, there are differences between internal systems and something like Twitter on the public internet. I assume that Twitter is a system under constant attack. What happens when the next log4shell level vulnerabilit…

> I've likened firing most of the engineers to taking your hands off the wheel in the car. It won't crash immediately, but it doesn't mean the car can go driverless.

This is an excellently apt analogy, in light of Twitter's new owner.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#806

Post by mlindner »

no_op wrote: post #648
thefz wrote: post #515

Earlier quoted context omitted.

> "Anyone who actually writes software, please report to the 10th floor at 2 pm today. Before doing so, please email a bullet point summary of what your code commands have achieved in the past ~6 months, along with up to 10 screenshots of the most salient lines of code" Actual quote. Anyone using the term "code commands" comes out a little detached from programming reality, let alone the rest of this request, it is o…

"Code commands" is very plausibly an autocomplete flub of what was supposed to be "code commits." When I type "code comm" my iPhone offers up "commands" as the completion. I've seen a lot of mockery of this request, but I suspect people aren't considering the wide variance in employee quality that can exist within a mismanaged organization. What Musk was asking for here wouldn't be a good way to evaluate skilled, con…

Thanks for the note about autocomplete. That explains it very well what he actually meant.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#807

Post by roguas »

jiggawatts wrote: post #723
karpathy wrote: post #539

Earlier quoted context omitted.

Elon also understands deep neural nets a lot more than I think people imagine. He starts with good intuitions and mental models, but also actively asks for technical deep dives, and has very good retention. E.g. I recall teaching him about our use of focal loss in contrast to binary cross-entropy for the object detection neural net (I said it had given us a 5% bump and he asked to know more) and he understood how it…

I think what upsets a lot of the Silicon Valley types here on HN is that people just like them are being called on their bullshit and fired en-masse for it. That has got to be uncomfortable. You know the old saying: "It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It." That's what's happening here. That engineer that was humiliated publicly for defending a slow-…

Every org has slow-as-molasses, badly designed, illogical components. It is about ratio.

Time will tell. I don't think Musk/twitter's case will set any precedent. He is too much of a character to provide broad meaningful insights into industry. Also he has accumulated a list of failures which are rarely mentioned.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#808

Post by pclmulqdq »

adam_arthur wrote: post #759
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

Well, WhatsApp had ~50 employees and Instagram around ~15 when FB acquired them, and they were around the same order of magnitude of complexity as Twitter. The only concern Id have is that by having so many people, your design probably comes to rely on them whereas a smaller team would be forced to make the system easier to maintain. Personally, if I were Elon, I’d build an entirely new backend and point the clients…

A lot of people who don't understand why Twitter owns two datacenters point to "complexity" as an argument and completely disregard scale. It turns out that massive scale adds a lot of complexity to a system, particularly around many-to-many pubsub systems (like social media). It also means a lot of features, like compliance with government regulations around the world.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#809

Post by animitronix »

goosedragons wrote: post #700
fnord123 wrote: post #472

Earlier quoted context omitted.

Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?

They had over a thousand moderators. So maybe your estimate of how many people is required is a bit off.

And they absolutely sucked at it. Good riddance to a useless cost center.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#810

Post by thinkmcfly »

donkeyd wrote: post #372
resonious wrote: post #340

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat. I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually…

> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…

We are entering the windows ME era of Twitter
Post reply on HN