Earlier quoted context omitted.
Judging by some of the old patents he's filed [1], I'd guess he has at least a decent understanding of the tech involved. Probably less so, when it comes to the details of more modern distributed systems, but I also wouldn't be surprised if he's spent some effort towards all that as well - he's been working in/around pretty cutting edge tech for quite a while. Could he sit down and code it himself? probably not, but…
Musk was fired from PayPal because he wanted to replace all the Unix servers with Windows.
Why Twitter didn’t go down: From a real Twitter SRE
321–330 of 1001 posts
Re: Why Twitter didn’t go down: From a real Twitter SRE
#322Earlier quoted context omitted.
Because politics and ideologues are so prevalent and everyone's attaching 'the other' (both sides) for the own. And of course people in 'important' roles who've been laid off are going to say the company is doomed, these are the probably the worst source to go off. They're not going to say "oh yeah I didn't do much at all really, just bossed people around and spent my budget every year."
It's actually scary how many people, even engineers, put their reputation on the line saying Twitter wouldn't survive the weekend. It wasn't just Twitter employees. It's like a mass psychosis of some kind. It comes off as a kind of desperation, as though they need Elon to fail. Why? What's driving that response?
Was that a real thing those people were saying?
Genuinely asking, as I don't follow any of the former Twitter engineers on, well, Twitter, and if there were any such posts/articles on here I must have missed them.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#323Earlier quoted context omitted.
> Twitter grew 3x on the headcount front There were multiple executives making $10m/yr+ There were board members There were shareholders Why did all of them not stop this headcount increase if it's as easily reduced as "too much headcount bad, smaller headcount good"? These are paid professionals who are supposedly wealthy, good at their jobs, smart, informed, etc. How can us commenters on HackerNews sit from our arm…
Companies are mismanaged into death all the time by groups of highly intelligent, previously successful people.
And the leads installed by the acquiring company seem to be highly intelligent - the company continues helicoptering in their brightest minds to fix this mess. It just so happens they are forced out every 12-24 months because it’s almost impossible to deliver anything there.
For example, last year they introduced SAFe. That caused an exodus which basically equaled a brain drain (still ongoing). Most people I valued for their straightforwardness and ability to cut through BS there have left or are leaving. The person pushing for SAFe introduction has resigned already.
The founders of that company have all left ofc, and are basically trying to restart the same company (or variations) with their own money. And their products are already picking up speed, becoming threats to the original one.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#324Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.
software, like everything, is subject to laws of physics I disagree; math would be a closer analogy. And indeed, arithmetic still works like it did a millenia ago. Closer to the present, I have binaries from the late 80s that still work today (and I use them semi-regularly.) Indeed, much of the impetus of the software industry seems to be to propagate the illusion that software somehow needs constant "maintenance" an…
Re: Why Twitter didn’t go down: From a real Twitter SRE
#325Earlier quoted context omitted.
It would be funny for Musk to share privately with friends. It is a deeply strange and inappropriate thing for the CEO of a billion dollar company to post publicly. Something is deeply wrong with Musk's decision making process.
He can get away with posting just about any silly thought because he has no board of directors to answer to and no one can fire him. Big difference when you are thinking of him as a CEO.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#326All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics. Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart. The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or…
Kevin Watson, who developed the avionics for Falcon 9 and Dragon and previously managed the Advanced Computer Systems and Technologies Group within the Autonomous Systems Division at NASA's Jet Propulsion laboratory: "Elon is brilliant. He’s involved in just about everything. He understands everything. If he asks you a question, you learn very quickly not to go give him a gut reaction.
He wants answers that get down to the fundamental laws of physics. One thing he understands really well is the physics of the rockets. He understands that like nobody else. The stuff I have seen him do in his head is crazy.
He can get in discussions about flying a satellite and whether we can make the right orbit and deliver Dragon at the same time and solve all these equations in real time. It’s amazing to watch the amount of knowledge he has accumulated over the years."
Re: Why Twitter didn’t go down: From a real Twitter SRE
#327Let don't kid ourselves here, even if Twitter crashes and needs a few weeks or even a few months to recover it doesn't matter. People who leaved will comeback again, after a year everything will be like nothing ever happened. And even if Twitter lost 50% of it users and revenues, it is still more profitable than have to pay the salary of 7000 people.
More profitable than what? It was and is losing money, plus now it also has some serious debt on top. With staff that actually know their shit (and a sane CEO), it may be possible to turn it profitable, but I don’t think it will happen as things currently stand.
Twitter revenue last year was around $5B.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#328Did the OP confirm Elon theory that most of the stuff is not needed? > For four of those years I was the sole SRE for the Cache team. There was a few before me, and the whole team I worked with, where a bunch came and went. But for four years I was the one responsible for automation, reliability and operations in the team. I designed and implemented most of the tools that are keeping it running so I think I’m qualifi…
One SRE, many SWE. Also have fun asking someone to be permanently oncall with one person on the team. The cache clusters size are also described here for anyone who wants a good technical read over speculation. https://www.usenix.org/system/files/osdi20-yang.pdf
> I designed and implemented most of the tools that are keeping it running so I think I’m qualified to talk about it.
Re: Why Twitter didn’t go down: From a real Twitter SRE
#329The most helpful thing to reflect on in these Twitter operational discussions is the difference between homeostasis and evolution. You can get rid of 80% of the work force and the existing homeostasis systems will keep things running smoothly despite known day-to-day chaos. Where you’re really going to run into trouble is inventing responses to novel chaos and gradually changing times.
We had an obituary for Fred Brooks on here just the other day. I'd suggest that his thesis in The Mythical Man-Month conflicts with your comment above (that reduction in staff count for a software project has a good correlation with the ability to maintain it / evolve it / innovate on top of it).
Re: Why Twitter didn’t go down: From a real Twitter SRE
#330Earlier quoted context omitted.
It's not goofballs. It's generally misaligned incentives. Managing a 10,000 org leads to better job prospects than a 1000 person org, than a 100 person org, than a pizza box team. Organizations tend to bloat. Random, rapid cuts might not be the fix here, but headcount was too high.
> Organizations tend to bloat. Say I'm an engineer. And I write code all day. And I get paid by a company to do it. And it blows up in production. And I tell my boss "code tends to blow up". Or I tell my boss "code tends to take longer than estimated to deliver". I wouldn't be given lots of promotions/bonuses with that outlook. We're talking about a Twitter CEO with $30m/yr+ in total compensation. There's no way his/…
> Or I tell my boss "code tends to take longer than estimated to deliver".
I've used variations of that phrase throughout my career. I'm very well paid, partly because I account for this and make certain that other stakeholders are aware.