Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter Is DDOSing Itself

sfba.social

Post reply on HN
Print view View original

631–640 of 1001 posts

Re: Twitter Is DDOSing Itself

#631

Post by spamizbad »

sigzero wrote: post #622
martincmartin wrote: post #373

Earlier quoted context omitted.

The people at Twitter who understood the system and could predict the side effects were all fired or left. My guess is Elon said "the site's too slow!" Engineers noticed that the home feed request was slow. They didn't understand how it worked, had no tools to profile it, and were given an unrealistic deadline to fix it. So about the only thing they could do was issue multiple, parallel requests and hope that at leas…

> The people at Twitter who understood the system and could predict the side effects were all fired or left. You have zero idea if that is true or not.

If it's not true that's actually more damning: The "experts" seem to frequently break it.

Re: Twitter Is DDOSing Itself

#632

Post by spamizbad »

andsoitis wrote: post #507
martincmartin wrote: post #373

Earlier quoted context omitted.

The people at Twitter who understood the system and could predict the side effects were all fired or left. My guess is Elon said "the site's too slow!" Engineers noticed that the home feed request was slow. They didn't understand how it worked, had no tools to profile it, and were given an unrealistic deadline to fix it. So about the only thing they could do was issue multiple, parallel requests and hope that at leas…

> The people at Twitter who understood the system and could predict the side effects were all fired or left. Are you saying the engineers who are now at Twitter don’t have the right skills?

They lack knowledge not skills. To my knowledge there is no "intuitively understand how my changes will impact a complex system without studying it" skill (unless that system is under a robust test suite, which Twitter is not)

Re: Twitter Is DDOSing Itself

#634

Post by dehrmann »

ikrenji wrote: post #519
lmeyerov wrote: post #329

Apparently their 5-year cloud contract failed to renew today So, there may be a much simpler explanation for why their new rate limits on regular users to can-barely-scroll levels, and how that has all sorts of unintended consequences they weren't ready for

google wouldn't just shut off services for a big customer like that...

A medium-sized customer making a point of not paying the bills to negotiate hard? If they come back, I'd put them on a pre-paid plan.

Re: Twitter Is DDOSing Itself

#635

Post by jayd16 »

martincmartin wrote: post #373
Topfi wrote: post #126

Speaking from very painful, personal experience, few things are more agitating than being forced to execute on something you fully know is a horrible idea, especially when you tried and failed to communicate this fact to the individual pushing you to go against your best judgement. Even more so when that person later loudly proclaims that they never made such a request, even when provided with written proof. I can of…

The people at Twitter who understood the system and could predict the side effects were all fired or left. My guess is Elon said "the site's too slow!" Engineers noticed that the home feed request was slow. They didn't understand how it worked, had no tools to profile it, and were given an unrealistic deadline to fix it. So about the only thing they could do was issue multiple, parallel requests and hope that at leas…

> The people at Twitter who understood the system

But this is Scaling-101 stuff. It's not some super complex or unique system going wrong. At least according to the article, it's a classic case of bad retry logic leading to a death spiral.

https://en.wikipedia.org/wiki/Thundering_herd_problem

Re: Twitter Is DDOSing Itself

#636

Post by tomcam »

Topfi wrote: post #126

Speaking from very painful, personal experience, few things are more agitating than being forced to execute on something you fully know is a horrible idea, especially when you tried and failed to communicate this fact to the individual pushing you to go against your best judgement. Even more so when that person later loudly proclaims that they never made such a request, even when provided with written proof. I can of…

How do you force someone to work at Twitter?

Re: Twitter Is DDOSing Itself

#637

Post by firesteelrain »

anders_p wrote: post #254
lr4444lr wrote: post #207

Earlier quoted context omitted.

I dunno. We are several months out after the major layoffs. Maybe some very recent bad decisions were made internally that pared back too far, but I think Musk has long since been proven correct that the core platform could function on a fraction of the workforce it had at the time of takeover.

>I think Musk has long since been proven correct that the core platform could function on a fraction of the workforce it had at the time of takeover. It REALLY sounds like you don't understand how any of this works. Tech products don't stop working when you fire most of the staff. But bugs stop being fixed and problems begin to add up, until a critical point is reached,m where the whole house of cards collapses. Thin…

Pretty sure parent does

Plus there’s always technical debt and even the ‘best’ engineers at Twitter made mistakes

Maybe if the architecture wasn’t so brittle and more easily testable by these engineering ‘gods’ then we wouldn’t have this problem

Re: Twitter Is DDOSing Itself

#638

Post by greenthrow »

humean1 wrote: post #542
martincmartin wrote: post #373

Earlier quoted context omitted.

The people at Twitter who understood the system and could predict the side effects were all fired or left. My guess is Elon said "the site's too slow!" Engineers noticed that the home feed request was slow. They didn't understand how it worked, had no tools to profile it, and were given an unrealistic deadline to fix it. So about the only thing they could do was issue multiple, parallel requests and hope that at leas…

> The people at Twitter who understood the system and could predict the side effects were all fired or left And never created sufficient testing.

You can't write tests for the changes someone might make in the future.

Re: Twitter Is DDOSing Itself

#640

Post by mrcwinn »

epistasis wrote: post #286
aeyes wrote: post #272

This bug is very unlikely to be the reason. The rate limiter on the server side is cheap and the frontend bug only gets triggered with the rate limit active. I have seen similar bugs in the systems I oversee because network libraries love to retry requests without sane limitations by default. But I never saw them make our rate limiters sweat. It's slightly more annoying when they hit an API which actually does some e…

One thing about having leadership that is known to lie about anything or everything, for any sort of imagined personal gain, is that the very concept of truth is destroyed. I agree that this is probably not the bug at the root of it all. But I also don't believe the story that Musk is selling for why he's in effect shutting down the site. But both could be true and I'm still thinking about other potential reasons, a…

This.

The media is flawed. Journalists are flawed. The Verge in particular has some atrocious coverage of tech.

But the problem with Elon is that in his world, we must rely on him for true information. As if the direct source is inherently true. He’s just as full of it as anyone else with interests.

Post reply on HN