Earlier quoted context omitted.
They lack knowledge not skills. To my knowledge there is no "intuitively understand how my changes will impact a complex system without studying it" skill (unless that system is under a robust test suite, which Twitter is not)
> knowledge there is no "intuitively understand how my changes will impact a complex system without studying it" skill (unless that system is under a robust test suite, which Twitter is not then how do they have confidence that anything works before they subject hundreds of millions of people around the world to system updates? that seems disrespectful to the user, if you asked me.
Twitter Is DDOSing Itself
801–810 of 1001 posts
Re: Twitter Is DDOSing Itself
#802Earlier quoted context omitted.
He did make the call to do that, did he not? He publicly stated it on Twitter.
Why does it matter that he made the call to do that? I get plenty of "do this, do that" requests from my manager. And if I break things, is it my manager's fault? I'm trying to understand your logic. Please, explain. Edit: here is my take on it + the most likely scenario of what's going to happen next. This was a bad deployment and such things are not unusual in software industry. In this specific case, there are zer…
Re: Twitter Is DDOSing Itself
#803Earlier quoted context omitted.
I’ll play the devils advocate here but frontend devs need to smarten up. This is basic error handling that should have been in place for years. Blocking tweets with 403 or whatever they chose shouldn’t trigger endless retries on short intervals.. ever!
Maybe the frontend team and QA were never made aware that 403 errors could be triggered like this.
Re: Twitter Is DDOSing Itself
#804Re: Twitter Is DDOSing Itself
#805Re: Twitter Is DDOSing Itself
#806Earlier quoted context omitted.
Devil's advocate here: did we consider that any such exponential back off goes out the window when users, faced with a non-working site, will just refresh the page therefor reseting the whole process?
The server load from that is negligible since those requests stop at the load balancer. On that note, the 10 requests/second in the post is also negligible for the same reason. Only requests that hit backend servers matter
Re: Twitter Is DDOSing Itself
#807Earlier quoted context omitted.
The people at Twitter who understood the system and could predict the side effects were all fired or left. My guess is Elon said "the site's too slow!" Engineers noticed that the home feed request was slow. They didn't understand how it worked, had no tools to profile it, and were given an unrealistic deadline to fix it. So about the only thing they could do was issue multiple, parallel requests and hope that at leas…
> The people at Twitter who understood the system and could predict the side effects were all fired or left. Are you saying the engineers who are now at Twitter don’t have the right skills?
Re: Twitter Is DDOSing Itself
#808It's an interesting theory but the DDoS predates the decision to disable anonymous access. In fact, that decision was taken in order to mitigate an ongoing DDoS[0][1]. So while it's possible that the questionnable web front end retry logic is not helping things, it's not the root cause. [0] https://twitter.com/elonmusk/status/1674865731136020505 > Temporary emergency measure. We were getting data pillaged so much tha…
Re: Twitter Is DDOSing Itself
#809I'm astounded it took this long to finally die. All those Twitter engineers who were harassed by Elmo's very online fan club, and were treated with such cruelty and disdain on their way out, clearly did an incredible job. Hats off to them. The curse of IT Operations: if you do everything right, management grifters never think you did anything at all. Why do I need you? Everything works fine! Most Internet platforms a…
Musk has a bachelor's degree in physics from the University of Pennsylvania: https://www.snopes.com/fact-check/musk-physics-degree/.
Further, while I share some criticism of the man, many very technical people from companies such as SpaceX and Tesla have come forward with public comments, praising him for "truly grasping the engineering" and "being involved in every technical design decision". Make of these what you will:
Kevin Watson, Falcon 9 avionics:
Elon is brilliant. He’s involved in just about everything. He understands everything. If he asks you a question, you learn very quickly not to go give him a gut reaction.
He wants answers that get down to the fundamental laws of physics. One thing he understands really well is the physics of the rockets. He understands that like nobody else. The stuff I have seen him do in his head is crazy.
He can get in discussions about flying a satellite and whether we can make the right orbit and deliver Dragon at the same time and solve all these equations in real time. It’s amazing to watch the amount of knowledge he has accumulated over the years.
Tom Mueller, SpaceX founding employee:
We’ll have, you know, a group of people sitting in a room, making a key decision. And everybody in that room will say, you know, basically, “We need to turn left,” and Elon will say “No, we’re gonna turn right.” You know, to put it in a metaphor. And that’s how he thinks. He’s like, “You guys are taking the easy way out; we need to take the hard way.”*
And, uh, I’ve seen that hurt us before, I’ve seen that fail, but I’ve also seen— where nobody thought it would work— it was the right decision. It was the harder way to do it, but in the end, it was the right thing.*
Garrett Reisman, engineer and former NASA astronaut:
What's really remarkable to me is the breadth of his knowledge. I mean I've met a lot of super super smart people but they're usually super super smart on one thing and he's able to have conversations with our top engineers about the software, and the most arcane aspects of that and then he'll turn to our manufacturing engineers and have discussions about some really esoteric welding process for some crazy alloy and he'll just go back and forth and his ability to do that across the different technologies that go into rockets cars and everything else he does.
Josh Boehm, former Head of Software QA at SpaceX:
Elon is both the Chief Executive Officer and Chief Technology Officer of SpaceX, so of course he does more than just ‘some very technical work’. He is integrally involved in the actual design and engineering of the rocket, and at least touches every other aspect of the business (but I would say the former takes up much more of his mental real estate). Elon is an engineer at heart, and that’s where and how he works best.
Along with many others.
Re: Twitter Is DDOSing Itself
#810Earlier quoted context omitted.
>The people at Twitter who understood the system and could predict the side effects were all fired or left Not necessarily. I’ve predicted bad outcomes for decisions in a few cases and been ignored but stuck around regardless. Mostly because I like my job and the goals of my organization even if it makes bad decisions. Of course to remain productive and improve my influence in future decision making it is absolutely…
> goals of my organization What do you think they are?
It’s a difficult balance: is in fact necessary to continually achieve but there are times where decisions focus exclusively, or at least too much, on . But we live in the real world, and sometimes that’s necessary. ( is not money though it has an impact on our financials) And also sometime people with a broader view see further than I do and those choices that seem wrong come around a few years later and it turns out is actually better off for it. It keeps me humble, skeptical of my own certainty even when it seems faultless.
But I’m also at a point now where people who pop up and start shouting ! or ! or something completely random like “Well how about ?” I can easily deal with: I go back, do a bit of the work I do, show it to the right people, and those shouts -disappear. Sometimes one gets through and it’s annoying, but whatever, nothing is perfects.
Of course the above vastly oversimplifies things. There are many, many more variables to juggle along the way. But I hope it gives a reasonable sense of things.
And I’m sure the “” style notation of things in my explanation makes it harder understand what I mean, but I value my privacy, hence the abstractions of the factors involved.