Twitter Is DDOSing Itself • Hacker News Board

Re: Twitter Is DDOSing Itself

#641

Post by vba616 » Sun, Jul 02, 2023, 2:30 AM UTC

vba616 wrote: post #618

tenpies wrote: post #425

Earlier quoted context omitted.

[flagged]

The major tech companies outsource stuff that matters - I mean, for 20 years at least - to contracted outside companies who pay 1/10th the amount to Americans in "flyover" country and even less to offshore employees. They do this because SWE prima donnas will not do an adequate job for some things no matter how much you pay them. Like, oh, say, cleaning the office microwave. So I guarantee you that you are overlookin…

P.S. "cleaning the office microwave" is sarcasm...a placeholder for an infinite collection of jobs, some programming and some not.

BPO = Business Process Outsourcing

Re: Twitter Is DDOSing Itself

#642

Post by nborwankar » Sun, Jul 02, 2023, 2:30 AM UTC

andsoitis wrote: post #507

martincmartin wrote: post #373

Earlier quoted context omitted.

The people at Twitter who understood the system and could predict the side effects were all fired or left. My guess is Elon said "the site's too slow!" Engineers noticed that the home feed request was slow. They didn't understand how it worked, had no tools to profile it, and were given an unrealistic deadline to fix it. So about the only thing they could do was issue multiple, parallel requests and hope that at leas…

> The people at Twitter who understood the system and could predict the side effects were all fired or left. Are you saying the engineers who are now at Twitter don’t have the right skills?

Alternately, if they have the “right skills” why is this happening? Clearly they don’t have some of the needed skills to prevent this. And a self-DDOS might be a good indication of that.

Re: Twitter Is DDOSing Itself

#643

Post by Solvency » Sun, Jul 02, 2023, 2:31 AM UTC

darkwraithcov wrote: post #78

Mizoguchi wrote: post #17

This kind of stuff is unavoidable with what's going on at Twitter. Infrastructure changes, platform changes and mass layoffs all at once. I'm actually impressed they haven't experienced more and longer outages.

Its a testament to how well designed and implemented the code base is, if the wheels havent started falling off yet (shockingly). I know we’re all supposed to shit on Twitter, but they had world class engineers working there before the mass layoffs and brain drain.

I can easily visualize a wooden wagon rolling along a rocky slow descent forever and all of the reasons its wheels would eventually break loose.

In the case of a platform like Twitter, aside from random acts of god and weather, aside from money, what are the sort of stuff that slowly breaks its wheels off over time such that, without staff, it inevitably stops working completely?

Re: Twitter Is DDOSing Itself

#644

Post by anonymoushn » Sun, Jul 02, 2023, 2:32 AM UTC

sigzero wrote: post #622

martincmartin wrote: post #373

Earlier quoted context omitted.

The people at Twitter who understood the system and could predict the side effects were all fired or left. My guess is Elon said "the site's too slow!" Engineers noticed that the home feed request was slow. They didn't understand how it worked, had no tools to profile it, and were given an unrealistic deadline to fix it. So about the only thing they could do was issue multiple, parallel requests and hope that at leas…

> The people at Twitter who understood the system and could predict the side effects were all fired or left. You have zero idea if that is true or not.

We can just read posts by former twitter staff engineers who say the same thing.

Re: Twitter Is DDOSing Itself

#645

Post by fwlr » Sun, Jul 02, 2023, 2:32 AM UTC

mananaysiempre wrote: post #323

fwlr wrote: post #283

Oh come on. This is console.log spam. This isn’t a “self-DDoS”.

No, look closely: the animation does not show the console, it shows the network pane. Each line is a (presumably unsuccessful) request to the Twitter backend. If a lot of people have the Twitter web frontend open and running in this state, it could in fact overload the backend.

Sure, I mis-spoke in saying it was console.log spam, that’s not specifically what’s happening. But let’s be real, if you’re getting a 429 response that is at most a cached hash table lookup per request, for a site has probably top 50 in the world in terms of “scaling to handle heavy request loads” for a decade.

There’s just no chance this specific thing is actually what’s causing issues for Twitter, it’s obviously a consequence of the heavy scraping and steps to stop the heavy scraping. It frustrates me to no end that smart technical software people, who can have intelligent discussions on not just code but a wide range of topics from nuclear powers to superconductors, suddenly lose their ability to have these discussions when a certain person is involved.

Re: Twitter Is DDOSing Itself

#646

Post by 95014_refugee » Sun, Jul 02, 2023, 2:34 AM UTC

andsoitis wrote: post #507

martincmartin wrote: post #373

Earlier quoted context omitted.

The people at Twitter who understood the system and could predict the side effects were all fired or left. My guess is Elon said "the site's too slow!" Engineers noticed that the home feed request was slow. They didn't understand how it worked, had no tools to profile it, and were given an unrealistic deadline to fix it. So about the only thing they could do was issue multiple, parallel requests and hope that at leas…

> The people at Twitter who understood the system and could predict the side effects were all fired or left. Are you saying the engineers who are now at Twitter don’t have the right skills?

William of Ockham would like a word.

Re: Twitter Is DDOSing Itself

#647

Post by nashashmi » Sun, Jul 02, 2023, 2:36 AM UTC

paxys wrote: post #572

tmpz22 wrote: post #413

Can we please recognize that this is a holiday weekend and Elon rammed a big release forcing his engineers to come in and iterate multiple patches over the last 12 hours.

Twitter Engineers have had over a year to get out and find another job. At this point the terms of their employment and the expectations from their boss are all crystal clear. Hard to sympathize with those who continue to stick around for whatever reason.

The activist mindset is not part of professionalism. If people chose to stick with their jobs, it is their professionalism. And if people chose not to leave, it is a sacrifice they made.

Any professional worth his salt would do the same.

Re: Twitter Is DDOSing Itself

#648

Post by chris_wot » Sun, Jul 02, 2023, 2:38 AM UTC

sigzero wrote: post #622

martincmartin wrote: post #373

Earlier quoted context omitted.

The people at Twitter who understood the system and could predict the side effects were all fired or left. My guess is Elon said "the site's too slow!" Engineers noticed that the home feed request was slow. They didn't understand how it worked, had no tools to profile it, and were given an unrealistic deadline to fix it. So about the only thing they could do was issue multiple, parallel requests and hope that at leas…

> The people at Twitter who understood the system and could predict the side effects were all fired or left. You have zero idea if that is true or not.

It’s likely. The number of people predicting degradation were wrong only on the length of time it took for the degradation to start occurring.

Re: Twitter Is DDOSing Itself

#649

Post by nashashmi » Sun, Jul 02, 2023, 2:39 AM UTC

tmpz22 wrote: post #413

Can we please recognize that this is a holiday weekend and Elon rammed a big release forcing his engineers to come in and iterate multiple patches over the last 12 hours.

The arrogance of the guy is so huge, hubris does not touch him. Thoughts are with the programmers.

What I’m more surprised about is how gum and shoestring the twitter engineering is now a days. They put in no emphasis on doing deep divides into the code base and instead opt to do the simplest shortest fix. And it causes problems.

Re: Twitter Is DDOSing Itself

#650

Post by the-rc » Sun, Jul 02, 2023, 2:40 AM UTC

goalieca wrote: post #516

Topfi wrote: post #126

Speaking from very painful, personal experience, few things are more agitating than being forced to execute on something you fully know is a horrible idea, especially when you tried and failed to communicate this fact to the individual pushing you to go against your best judgement. Even more so when that person later loudly proclaims that they never made such a request, even when provided with written proof. I can of…

I’ll play the devils advocate here but frontend devs need to smarten up. This is basic error handling that should have been in place for years. Blocking tweets with 403 or whatever they chose shouldn’t trigger endless retries on short intervals.. ever!

Not just frontend developers. The backend should serve a 429 or 503 error, complete with a Retry-After header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Re...

That would give the server side more control over the retrying logic (when the header is properly interpreted). I'm surprised Elon hasn't implemented this himself.