Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Twitter didn’t go down: From a real Twitter SRE

matthewtejo.substack.com

Post reply on HN
Print view View original

961–970 of 1001 posts

Re: Why Twitter didn’t go down: From a real Twitter SRE

#961

Post by henrikschroder »

kortilla wrote: post #568
henrikschroder wrote: post #406

Earlier quoted context omitted.

Your reply is indistinguishable from a religious person exclaiming that "God is omniscient and moves in mysterious ways!" Stop worshipping billionaires.

> Stop worshipping billionaires. This is a vapid over-generalization designed to dismiss any statements by attaching them to the wealth, not the person. Don’t do this. Nobody is talking about the Waltons, Buffet, Larry Ellison, etc. This is a discussion about Musk specifically and his history of operating companies.

The US has been formed religiously by various congregations that descend from Calvinism, which teaches that hard work earns you the grace of God. In modern times, this has morphed into Prosperity Gospel which teaches that if you are loved by God, you will become materially successful, as well as the flipside; if you are materially successful, you must therefore be loved by God.

The billionaire worship is simply the secular version of this faith. X made billions of dollars by owning a business that does Y, therefore X must be an expert on Y, and businesses in general, and is a genius, and is a morally superior person. Otherwise X wouldn't be a billionaire, because - horror of all horrors - the world is just, isn't it? It couldn't be the case that X just got lucky, can it? Surely, X has earned their billions, right?

Re: Why Twitter didn’t go down: From a real Twitter SRE

#962

Post by [deleted] »

karpathy wrote: post #539
memish wrote: post #326

Earlier quoted context omitted.

John Carmack, "Elon is definitely an engineer. He is deeply involved with technical decisions at spacex and Tesla. He doesn’t write code or do CAD today, but he is perfectly capable of doing so." Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon i…

Elon also understands deep neural nets a lot more than I think people imagine. He starts with good intuitions and mental models, but also actively asks for technical deep dives, and has very good retention. E.g. I recall teaching him about our use of focal loss in contrast to binary cross-entropy for the object detection neural net (I said it had given us a 5% bump and he asked to know more) and he understood how it…

[deleted]

Re: Why Twitter didn’t go down: From a real Twitter SRE

#963

Post by ggm »

housecarpenter wrote: post #771
ggm wrote: post #230

The article is good, and informative but a little odd in one respect. It uses air-quotes to introduce concepts like "rack" and "shelf" in a DC, but does O(logn) notation mid-flow. If you don't know what a rack is, how are you meant to know what the Order of scaling function means? Thats a highly computer sciences specific notation, and if you grok O(n) you know what a rack, a host, a DC is.

I get your point that both of these are specialized notations/terminologies, but it's entirely possible for somebody to understand algorithmic complexity without knowing what "a rack, a host, a DC" is, there's no inherent connection between the two. As somebody whose interests tend to be more theoretical than practical, myself prior to reading the article would be an example.

As the OP said "I was trying to write for a more general audience."

Re: Why Twitter didn’t go down: From a real Twitter SRE

#964

Post by edmundsauto »

WalterBright wrote: post #333
edmundsauto wrote: post #277

Earlier quoted context omitted.

Automotive and aerospace are not that similar to social media. People buying into the vision of "get the planet off fossil fuels for transport" and "get this species to Mars" are probably willing to make sacrifices that people working on social media are not. It's the Halo Effect fallacy to think competence in one field automatically translates to another. Especially when the founder in question has displayed increas…

Automotive and aerospace are not similar to each other, either. I don't know any other outfit that was successful at both. > It's the Halo Effect fallacy to think competence in one field automatically translates to another I didn't say it was. I was responding the notion that Musk blundered into success at Tesla and SpaceX.

There's a lot to unpack here, for one do we consider those two enterprises to be successes? It seems too early to tell - they do appear to be influential, but the C64 and Palm Pilot were influential without being successful. It's also not clear if they are long term successful (I think Tesla will be, but SpaceX is very much currently dependent on government funding). Finally, it's not clear whether Musk was a critical driver of success - evaluating his contributions based on the outcome of a company is basically resulting.

Look, I don't know if Elon is a genius or an opportunistic parasite with really good PR. It seems unlikely if we ever will know that. What I object to is people pointing at his ultimate financial success and crediting him with the current result of 2 big companies whose future is very much not determined.

When I look at his process from this ant's perspective, I think he is an abusive unstable individual who takes credit for everyone's work and lies a lot. He also flip-flops depending on the wind. Is that success? Not based on my personal values. Have his companies accomplished a lot? Some of them, absolutely.

The definitions and evidence matter a lot, and I personally don't think any of us are qualified to make blanket statements based on incomplete outcomes. Further, I don't think his other companies that require primarily good engineering are very relevant to inherently people problems, like Twitter. My evaluation of how Musk handles people problems is that he is very bad at them, and I anchor my prediction about his Twitter leadership based on that.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#965

Post by twblalock »

cm2187 wrote: post #449
twblalock wrote: post #409

Earlier quoted context omitted.

It’s easy to think it’s bloat at a steady state. When something important goes down and nobody knows how to fix it, it looks different.

That being said it's not like twitter is a massively complex product with lots of different features. I can imagine you could keep it running with a skeletton team. Liasing with ads buyers excepted.

Even "simple" things become complex when they need to scale to billions of users around the world, handle a high rate of traffic, deal professionally and legally with all the different jurisdictions they operate in and all the advertisers they serve, etc.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#966

Post by themadturk »

polynomial wrote: post #874
donkeyd wrote: post #830

Earlier quoted context omitted.

Good point. I know this as the Mack Truck Theory. For the project I'm working on right now, there's a couple of incredibly valuable people that would cause a pretty significant issue if they disappeared.

Sensitive workplaces (which seems like most these days) have taken to calling this the Lottery Factor (as in team members quit bc they hit the lottery) to spare more delicate types the pain of imagining their peers run over in traffic accidents.

Almost lost a co-worker when he got hit by a bus five years ago. He's back at work full time now, with a substantial settlement from the transit agency (nice amount of f** you money). He wasn't an essential employee, but was important to our team in many ways. Fortunately he was always willing to joke about his accident. He'd survived, after all!

Re: Why Twitter didn’t go down: From a real Twitter SRE

#967

Post by Tiktaalik »

h11h wrote: post #493
minhazm wrote: post #373

Earlier quoted context omitted.

Many people who have worked with Musk have shared similar sentiments in interviews. But it seems that people just refuse to believe any of it. People think that there's no way it's possible for someone to be that deeply technical and be a CEO of multiple companies at the same time. I've talked to people about it and they straight up refuse to believe it saying that it's impossible and that any evidence of him being t…

I'm inclined to believe this is true. The problem is that Twitter's challenges are social/political, not technical, and Musk has demonstrated little competence in this area.

Another area where Musk's "brilliance" has faltered is in his transportation ideas and there's a connection here to twitter in that the challenges here again are not technical, but rather political around land use.

Brute forcing a problem with better technology not always the actual solution when technical problems aren't actually the problem.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#968

Post by donkeyd »

celim307 wrote: post #868
donkeyd wrote: post #830

Earlier quoted context omitted.

Good point. I know this as the Mack Truck Theory. For the project I'm working on right now, there's a couple of incredibly valuable people that would cause a pretty significant issue if they disappeared.

Sounds like those people are in strong negotiation positions

One of them just got promoted quite nicely, so I guess you're right!

Re: Why Twitter didn’t go down: From a real Twitter SRE

#969

Post by AtlasBarfed »

usrusr wrote: post #691
AtlasBarfed wrote: post #164

Twitter is a lot of drama, but ultimately this is a common private equity play: 1) take private 2) fire/layoff until things break, patch up / rehire with cheaper labor. Repeat as necessary 3) spit shine/repackage what's left with a theoretically more appealing balance sheet. 4) resell to another sucker...uh, buyer, or take public again. Fundamentally Musk bought 44 billion for 5 billion of annual revenue, and presuma…

4) who'd that sucker be? How much hubris would it take to sell investors on the idea of salvaging a failing company that "the guy who killed the internal combustion engine" and "the guy who made rockets land on their tail" couldn't make profitable? If it comes to selling off a failure, Musk's reputation will be quite a dealbreaker. The days of whatever happens, if all else fails we can still sell to Yahoo are long go…

If Elon holds the line on advertising and drops costs such that Twitter turns two billion in profit, they yeah the narrative in the business circles will be that Elon trimmed the fat and turned twitter into a viable property and someone with cash will buy it. I mean, Microsoft paid 8 billion for github! They bought Skype for 8.5 billion (probably like 12-20 billion these days), and THAT was after ebay bought skype for 2.6 billion (and none of the source code rights) in what was considered a vast overpay.

Re: Why Twitter didn’t go down: From a real Twitter SRE

#970

Post by sleight42 »

throwaway892238 wrote: post #130

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…

"Outages": this is an enormous ellipsis.

* Power outages and general acts of God

* Resource utilization

How do your databases perform when their CPUs are near capacity? Or disks? Or I/O? I've seen Postgres do some "weird s%$#": where query times don't go exponential but they go hockey stick.

* Fan-out and fan-in

These can peg CPU, RAM, I/O. Peg any one of these and you're in trouble. Even run close to capacity for any one of these and you're liable to experience heisenbugs. Troublesome fan-out and fan-in can sometimes be a result of...

* Unintended consequences

The engineering decision made months or years ago may have been perfectly legitimate for the knowledge available at the time. However, we live in a volatile, uncertain, complex, and ambiguous (VUCA) world; conditions change. If your inputs deviate qualitatively or quantitatively significantly, you risk resource utilization issues or, possibly, good ol' fashioned application errors.

"No battle plan survives contact with the enemy." -- Stormin' Norman

Same with software systems. They're living entities that can only maintain homeostasis so long as their environment remains predictable within norms. Deviate far enough from that and boom.

Post reply on HN