Earlier quoted context omitted.
Yeah it's dancing around the question: Was Musk right? All signs so far are pointing to, yes . MBA's will be studying this for years.
> All signs so far are pointing to, yes which signs? They launched a new feature (blue checks for $8) and had to turn it off immediately because it was bleeding money and ruining the platform and they have less ad revenue booked for next year than they had at the same time last time. I don't think one should judge the new twitter course yet, but "well, the site is still up" is a very bad measure of success.
Why Twitter didn’t go down: From a real Twitter SRE
511–520 of 1001 posts
Re: Why Twitter didn’t go down: From a real Twitter SRE
#512Earlier quoted context omitted.
> the article might actually persuade me that it was all bloat First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject. Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of thes…
Twitter had 7,500 employees. most of the roles you mention (security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, scrum master) are not bloat. So the question is what are the other 7000 people doing?
Re: Why Twitter didn’t go down: From a real Twitter SRE
#513Earlier quoted context omitted.
> He proved over and over again. [Narrator voice] He did not.
Did SpaceX failed for bankruptcy or something? Did Tesla not change the whole industry?
Re: Why Twitter didn’t go down: From a real Twitter SRE
#514All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…
And he hates bloated inefficient teams. His decrees on meetings are infamous. Tripling the team at Twitter implies a lot of internal politics, fiefdoms, communication overhead, and generally a lot of headless chickens running around. There's no nice way to fix such a team. A sledge hammer is one way to fix it and obviously he likes getting results quickly.
So, the notion of laying off most of that team was a foregone conclusion. The notion that a lot of the better people would get upset about that and leave as well is also highly predictable. What's left is a team with some gaps but also a lot of breathing room. And he can always lure key people back in by throwing money at them.
Simple plan. It might actually work. At the cost of a bit of drama, temporary instability, and lots of free publicity. Exactly his style. Cringe worthy and effective. I can see the logic here.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#515All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…
John Carmack, "Elon is definitely an engineer. He is deeply involved with technical decisions at spacex and Tesla. He doesn’t write code or do CAD today, but he is perfectly capable of doing so." Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon i…
Actual quote. Anyone using the term "code commands" comes out a little detached from programming reality, let alone the rest of this request, it is out of a Dilbert strip.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#516Earlier quoted context omitted.
And Twitter's problem are nowhere near technological. The site needed to make more money, not reengineer the whole thing while advertisers are fleeing because Trump is back on on a whim!
So should the platform be guided by advertisers? Especially one that’s apparently the de facto public square?
Re: Why Twitter didn’t go down: From a real Twitter SRE
#517I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the goo…
The reality is most huge companies are majority bloat. The hiring numbers are also in part crap that goes into Series X Raise pitch decks. Oftentimes a lot of the new bloat pisses off competent people, because their work doesn't actually get less, it becomes more. Not only do they have to now nanny people that are often not actually competent in their job, they just happened to go through the coding interview with wh…
German here. I think this actually is a huge part of the success of the famous Mittelstand - all that institutional knowledge these people have is extremely valuable. It's not just basic stuff like "know time tracking, billing and other admin systems and internal processes", but also the stuff that really can speed up your work: whom to ask on the "kurzer Dienstweg" aka short-circuitting bureaucracy when needed, personal relationships with people in other departments on whose knowledge you rely (it's one thing if you get a random email asking for some shit from someone you don't know, but I'll always find some minutes to help out someone who has helped me out in the past), all the domain-specific knowledge about the precise needs and desires of your customers...
Attrition is bad for a company as a whole, the problem is US-centric capitalism cannot quantify that impact (and it doesn't want to, given that attrition-related problems are long-term issues with years of time to impact), and so there is no KPI for leadership other than attrition rate itself.
The only problem is that over the last years, employers' mindset has shifted from regular wage raises to paying the bare minimum which makes changing jobs every few years a virtual requirement for employees to get raises, and so we are already seeing the first glimpses of US employment culture and its issues cropping up.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#518Earlier quoted context omitted.
Quoted post unavailable.
Look at his claim that Twitter is slow in India because of poorly batched backend calls [1]. He's simply regurgitating what he thinks he heard from others without really understanding how anything works. He knows less about software architecture than an intern picked at random. 1. https://mobile.twitter.com/elonmusk/status/15921762028730859...
a) Twitter is atrociously slow for something that displays a mere kilobyte of text at a time.
b) Much of that is because of hundreds of client-server JavaScript requests on first load. Not technically RPCs, but just as slow and looking nearly identical on the network.
c) Most of the rest is because of a thousand microservices chatting away at each other. Not technically client-to-server, but still taking significant time.
Elon accidentally conflated the network path of (b) with the performance issues caused by (c).
They're both bad! They both need fixing! The fact that Musk mixed up doesn't matter.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#519I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.
>What exactly where these thousands of employees actually doing They had wine on tap.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#520From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running. The biggest thing that brings…
At least for expired certs, most people have learned the hard way just how bad that is, and either implemented automated renewal (thank to heavens for cert-manager, LetsEncrypt, AWS ACM and friends) or where that doesn't work (MS AD...) monitoring.