Reddit needs to implement this if they want to get to the next level.
Why Twitter didn’t go down: From a real Twitter SRE
971–980 of 1001 posts
Re: Why Twitter didn’t go down: From a real Twitter SRE
#972Earlier quoted context omitted.
John Carmack, "Elon is definitely an engineer. He is deeply involved with technical decisions at spacex and Tesla. He doesn’t write code or do CAD today, but he is perfectly capable of doing so." Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon i…
Elon also understands deep neural nets a lot more than I think people imagine. He starts with good intuitions and mental models, but also actively asks for technical deep dives, and has very good retention. E.g. I recall teaching him about our use of focal loss in contrast to binary cross-entropy for the object detection neural net (I said it had given us a 5% bump and he asked to know more) and he understood how it…
Re: Why Twitter didn’t go down: From a real Twitter SRE
#973Earlier quoted context omitted.
Whenever clients complain about those costs and efforts, i tell them it's just like their car. Your car is working perfectly fine so why should you pay for maintenance?
software doesn't break down from heat. An app I write today will run until the hardware dies. I have a palm_os app I wrote in 1998 that still runs perfectly.
How many SSL certificates (internal or external) need re-issuing per month? Some of that can be automated, but in an organization as large as complex as Twitter some will be bespoke and manual, and a code freeze won't stop the clock.
How many new CVEs per month apply to Twitter's services and tooling? How many race conditions or other bugs are lurking, just waiting for the right time or traffic pattern to emerge? Twitter can't freeze inbound traffic without dire consequences.
Twitter is like your car, except that it's always running.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#974Earlier quoted context omitted.
Good point. I know this as the Mack Truck Theory. For the project I'm working on right now, there's a couple of incredibly valuable people that would cause a pretty significant issue if they disappeared.
Sounds like those people are in strong negotiation positions
You'd be surprised, but that's not necessarily the case.
One of my friends was such a person in a shoe making company as a designer. Instead of giving her a raise, they fired her.
Cue them re-hiring her a month or two later after they found out the hard way that the less experience subordinate really couldn't handle the job of the two of them on their own.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#975Earlier quoted context omitted.
Quoted post unavailable.
Literally his first two companies, Zip2 (CTO) and x.com (CEO), were pure software companies? Another company he founded, SpaceX, is the first space launch provider to ever achieve propulsive landing and re-use of an orbital-class rocket, a feat which includes some of the hardest software engineering you can do. Another company he runs, Tesla, is famous for being one of the few car companies that makes software for th…
Not to be dismissive of landing an orbital rocket, but that's mostly a feat of aerospace engineering and control theory. On the software engineering side of things for SpaceX I think their work around CFD would be the more impressive feat from a computer science perspective.
Elon just made sweeping changes that cratered their ad revenue and took an axe to engineering teams all over the company. He did this in the lead up to the holiday season which has been the most profitable part of the year with advertisers pouring a significant portion of their yearly ad budgets into ad buys. Even if he was 100% correct that all of this was bloat, it was a monumentally stupid idea to make these changes now of all times. I would have expected him to at least have enough of an understanding of software engineering to know that deprecating systems and cutting out bloat needs to happen carefully with the talent onhand to be able to back out changes and manage the transition seamlessly. Fundamentally the risks involved are more financially impactful than the price of keeping headcount around for another year to do it safely.
Twitter hasn't even had a major incident yet and just the revenue loss from advertisers pausing ad campaigns probably already offsets all the savings they were hoping for in 4 - 6 months once the layoffs finally start bringing down staff costs.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#976Earlier quoted context omitted.
The headcount at WhatsApp in 2013 was somewhere between 50-100, at which time they were servicing approx 400m MAU, which is more than users than Twitter has been able to boast for most of their existence. Coincidentally, in 2013 SpaceX was just starting to provide commerical launch capacity, at which point I think they too had Not surprised Elon Musk thinks he can run twitter with a skeleton crew.
1) And what was their uptime in 2013? How did uptime change as the service grew in popularity? 2) WhatsApp does not support the type of public broadcasts done at Twitter, and due to its e2ee doesn't require much human moderation.
2) Public broadcasts make a lot of things easier because that means more of your workload is relatively straightforward caching (as evidenced by this blog post).
Re: Why Twitter didn’t go down: From a real Twitter SRE
#977Twitter is a lot of drama, but ultimately this is a common private equity play: 1) take private 2) fire/layoff until things break, patch up / rehire with cheaper labor. Repeat as necessary 3) spit shine/repackage what's left with a theoretically more appealing balance sheet. 4) resell to another sucker...uh, buyer, or take public again. Fundamentally Musk bought 44 billion for 5 billion of annual revenue, and presuma…
Chinese developers are famous for making apps serving 1.4 billion people, with at most 1/3 salary of US, and the willingness to do 996 a whole year around. I would be surprised that no company in the west take advanced of that.
2000 miles and a language barrier do not help things.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#978Earlier quoted context omitted.
The other side of that is browsers. Even if you don’t change your code, the platform people are running your code in changes, automatically in many cases. New JS or CSS behavior in next safari or chrome? You need to patch/push to accommodate running environments that are outside your control.
The old space jam site worked for ages, and would still work if they hadn’t taken it down. The web is pretty good about keeping backwards compatibility.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#979Earlier quoted context omitted.
Quoted post unavailable.
Is this based on personal experience, or are you going by some well known publicly available stories about Musk? I've heard pretty much every possible opinion about the guy, but the ones that impress me the most are that he loves to go deep into technical topics with people who know their stuff, regardless of their credentials. That, to me, is a sign of a man with deep humility and desire to learn. At the same time,…
It is based on the fact that be publicly fired very competent people on Twitter for contradicting or criticizing him.
https://timesofindia.indiatimes.com/gadgets-news/this-is-wha...
https://www.businessinsider.com/tesla-elon-musk-ruthlessly-f...
He also publicly mocked all the people he fired
https://www.ndtv.com/world-news/elon-musks-dig-at-sacked-emp...
Re: Why Twitter didn’t go down: From a real Twitter SRE
#980Earlier quoted context omitted.
It would be funny for Musk to share privately with friends. It is a deeply strange and inappropriate thing for the CEO of a billion dollar company to post publicly. Something is deeply wrong with Musk's decision making process.
His process tree has only produced private and reusable rockets, space internet, electric vehicles, and hundreds of billions in value. But he makes crude jokes like 90% of the men on the whole planet. Clearly there's something deeply wrong.