Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter Is DDOSing Itself

sfba.social

Post reply on HN
Print view View original

581–590 of 1001 posts

Re: Twitter Is DDOSing Itself

#581

Post by in_cahoots »

williamsmj wrote: post #543
walrus01 wrote: post #437

Earlier quoted context omitted.

> The people at Twitter who understood the system and could predict the side effects were all fired or left. This is like a case study in what happens when you fire everyone except the sycophants and yes-men. I only feel sorry for remaining non-yes-men twitter employees who might still be there because for whatever personal reasons they're in a precarious economic situation where they can't quit (H1B?) or are tied to…

> This is like a case study in what happens when you fire everyone except the sycophants and yes-men. The initial and biggest waves of layoffs last year were of people who hadn’t yet had a chance to demonstrate whether they were or were not sycophants. They were essentially random.

I don’t think that’s true at all. Elon allowed any employee who wanted to quit with three months’ severance. Anyone who wasn’t a true believer and had options could have quit.

Re: Twitter Is DDOSing Itself

#582

Post by whatever1 »

humean1 wrote: post #542
martincmartin wrote: post #373

Earlier quoted context omitted.

The people at Twitter who understood the system and could predict the side effects were all fired or left. My guess is Elon said "the site's too slow!" Engineers noticed that the home feed request was slow. They didn't understand how it worked, had no tools to profile it, and were given an unrealistic deadline to fix it. So about the only thing they could do was issue multiple, parallel requests and hope that at leas…

> The people at Twitter who understood the system and could predict the side effects were all fired or left And never created sufficient testing.

There is never sufficient testing. If there is, then you have a formal proof for your algorithm.

Re: Twitter Is DDOSing Itself

#584

Post by BirdieNZ »

tbrownaw wrote: post #465
praisewhitey wrote: post #355

Earlier quoted context omitted.

>Speaking from very painful, personal experience, few things are more agitating than being forced to execute on something you fully know is a horrible idea, especially when you tried and failed to communicate this fact to the individual pushing you to go against your best judgement. In this case the horrible idea is being forced to push changes to production at a moments notice

> push changes to production at a moments notice Isn't that one of the DORA metrics?

Yes, but so is Change Failure Rate (and a good team will use feature flagging anyway for gradual rollouts, so you don't roll out a DDoSing broken feature to 100% of users, just a tiny subset).

Re: Twitter Is DDOSing Itself

#585

Post by cratermoon »

wongarsu wrote: post #544
jlund-molfese wrote: post #526

Earlier quoted context omitted.

I’d bet (not that much, but like $20) that someone has a .ifClientError() or if responseStatus === 4xx somewhere. If you’ve never had to handle authorization in a particular area, it might have been safe to assume that any 4xx error should have been retried when the code was originally written and someone didn’t write them all out

But wouldn't responseStatus === 4xx indicate that the problem is on your end and retrying is unlikely to fix the issue. A 5xx is worth a retry, a 4xx should imho just produce an error message. And even if you do retry, exponential backoff has been the standard for a long time (and is mentioned by the Twitter API documentation as a good solution to 429 responses)

There are libraries, usually under the general heading of "circuit breaker", that handle 429 and other reasons to retry in a sane manner. I'm not a JS expert but I believe either yammer/circuit-breaker-js or nodeshift/opossum would work in the browser. Even a hand-coded exponential backoff with jitter is simple enough to do for most cases.

Re: Twitter Is DDOSing Itself

#586

Post by sanderjd »

epistasis wrote: post #286
aeyes wrote: post #272

This bug is very unlikely to be the reason. The rate limiter on the server side is cheap and the frontend bug only gets triggered with the rate limit active. I have seen similar bugs in the systems I oversee because network libraries love to retry requests without sane limitations by default. But I never saw them make our rate limiters sweat. It's slightly more annoying when they hit an API which actually does some e…

One thing about having leadership that is known to lie about anything or everything, for any sort of imagined personal gain, is that the very concept of truth is destroyed. I agree that this is probably not the bug at the root of it all. But I also don't believe the story that Musk is selling for why he's in effect shutting down the site. But both could be true and I'm still thinking about other potential reasons, a…

I know this is getting too political so I'll deserve the downvotes, but totally ignoring anything else, this is why I fundamentally could not and cannot stand Donald Trump. This is his playbook too, just constant willful dishonesty and distortion.

Re: Twitter Is DDOSing Itself

#588

Post by paxys »

buitreVirtual wrote: post #579
paxys wrote: post #572

Earlier quoted context omitted.

Twitter Engineers have had over a year to get out and find another job. At this point the terms of their employment and the expectations from their boss are all crystal clear. Hard to sympathize with those who continue to stick around for whatever reason.

In this job market, it's hard. And many engineers are on H1B visas so it's even harder to jump ship.

If they truly cannot get another job despite trying hard for 15 months then really they should be thanking Musk for giving them a steady paycheck when no one else in the industry will.

Re: Twitter Is DDOSing Itself

#589

Post by cratermoon »

lrvick wrote: post #340
Topfi wrote: post #126

Speaking from very painful, personal experience, few things are more agitating than being forced to execute on something you fully know is a horrible idea, especially when you tried and failed to communicate this fact to the individual pushing you to go against your best judgement. Even more so when that person later loudly proclaims that they never made such a request, even when provided with written proof. I can of…

[flagged]

It's blunt, but true: check your privilege.

Re: Twitter Is DDOSing Itself

#590

Post by wpietri »

tadfisher wrote: post #515
andsoitis wrote: post #507

Earlier quoted context omitted.

> The people at Twitter who understood the system and could predict the side effects were all fired or left. Are you saying the engineers who are now at Twitter don’t have the right skills?

That's a non-sequitir. You can be at the top of your field and not completely understand a complex system. Also, the right people may not have even been involved in the implementation of this "feature".

For sure. I'd add that even super-geniuses get overwhelmed when asked to do too many things at once. Kudos to whoever's still left at Twitter; the current erratic decay is better than I would have expected after this long. But when maintenance and improvement are deferred long enough, eventually you reach the point where on average solving 1 problem creates >1.0 problems.
Post reply on HN