Live data from Hacker News

Twitter Is DDOSing Itself

sfba.social

701–710 of 1001 posts

Re: Twitter Is DDOSing Itself

#701
post #676

Earlier quoted context omitted.

> The people at Twitter who understood the system and could predict the side effects were all fired or left. You have zero idea if that is true or not.

You might be right; the person you're replying to might have zero idea. Now me... I know someone personally who was a senior exec for Twitter's software team, who left after Elon's purge. He left because all the people who understood the system and could predict the side effects were fired or left. He'd been with companies going through death spirals before, and had no interest in being involved with another one. So,…

[flagged]

Re: Twitter Is DDOSing Itself

#702
post #597
post #591

Earlier quoted context omitted.

a company is made of: _____

The board of directors, rich shareholders and the C-Suite team. Incidentally they also employ powerless servants who are interchangeable and disposable.

Modern Western capitalism is pretty similar to the Soviet Union or China's.

"State capitalism" and "monopoly capitalism" (the bad kind according to socialists) are about as different as Coke and Pepsi.

Ultimately the "owners" in practice are people you never heard of that work for (government) pension funds and index funds. The billionaires, founders, and celebrities are mostly a sideshow. That's not a denial of wealth inequality, just as officially socialist countries have.

But nearly all big companies have no owner in a top hat running the show, just faceless committees, responsible for buying everything - that company and all its competitors. Bureaucrats, apparatchiks, public or private, it's much the same.

If the C-suite and directors of huge companies are generally parasites, it's because robber barons and corporate raiders are the exception today. People whose job is to invest trillions of dollars by simply buying everything for sale have the least control imaginable over company managers. The current situation reflects classic criticisms of socialism, but it won anyway.

Nearly every source of information on public company stocks in the US has a figure for "institutional ownership". If a company is of any size, and is a real business, this is usually a high % - this is a reality check if what I'm writing sounds outlandish because nobody talks about anything but Twitter around here.

Re: Twitter Is DDOSing Itself

#703
post #272

This bug is very unlikely to be the reason. The rate limiter on the server side is cheap and the frontend bug only gets triggered with the rate limit active. I have seen similar bugs in the systems I oversee because network libraries love to retry requests without sane limitations by default. But I never saw them make our rate limiters sweat. It's slightly more annoying when they hit an API which actually does some e…

Depends on the scale of the overall system. I have personally seen and attempted to mitigate degenerate cases where these retries overwhelmed the backend so much that the servers were falling behind in simply rejecting the requests. Infact it got so bad because of all those retries at multiple levels from upstream callers that requests were essentially timing out at the TCP buffer/queue before they could be processed…

It is unlikely that a system with the scale of Twitter implements the API rate limiter in the backend. Usually you'd do this as early as possible together with other WAF stuff.

If IPs or IP ranges get really annoying we block them on the network level.

Big public sites like Twitter obviously need to have this technology. Due to their political content they probably also need sophisticated DDoS protection.

Re: Twitter Is DDOSing Itself

#704

Earlier quoted context omitted.

> The people at Twitter who understood the system and could predict the side effects were all fired or left. You have zero idea if that is true or not.

About 80% of Twitter was laid off or quit. I think it's a reasonable supposition that a good number of those were critical personnel who felt they could get a better deal somewhere else.

[flagged]

Re: Twitter Is DDOSing Itself

#705
post #676

Earlier quoted context omitted.

You might be right; the person you're replying to might have zero idea. Now me... I know someone personally who was a senior exec for Twitter's software team, who left after Elon's purge. He left because all the people who understood the system and could predict the side effects were fired or left. He'd been with companies going through death spirals before, and had no interest in being involved with another one. So,…

[flagged]

"Something went wrong. Try reloading."

Re: Twitter Is DDOSing Itself

#707
post #635

Earlier quoted context omitted.

The people at Twitter who understood the system and could predict the side effects were all fired or left. My guess is Elon said "the site's too slow!" Engineers noticed that the home feed request was slow. They didn't understand how it worked, had no tools to profile it, and were given an unrealistic deadline to fix it. So about the only thing they could do was issue multiple, parallel requests and hope that at leas…

> The people at Twitter who understood the system But this is Scaling-101 stuff. It's not some super complex or unique system going wrong. At least according to the article, it's a classic case of bad retry logic leading to a death spiral. https://en.wikipedia.org/wiki/Thundering_herd_problem

[deleted]

Re: Twitter Is DDOSing Itself

#708

Earlier quoted context omitted.

What holiday is it?

Is this really easier than consulting a calendar? Its the weekend before the 4th of July, the independence day for USA - many employers give employees Monday and Tuesday off, creating a potential four day break during what is typically very good weather - that being said Twitter is presumably (still) an international company with people in all different holiday areas.

I searched "what holiday is it this weekend" on kagi and found nothing. I don't know what country GP is in and I haven't heard about any public holidays this weekend.

The "America is the default" is tiring. The US makes up ~ 5% of the population, and most of us do not default to American thinking.

Re: Twitter Is DDOSing Itself

#709

Earlier quoted context omitted.

> So about the only thing they could do was issue multiple, parallel requests and hope that at least one of them was fast. lol nobody would do this to solve this problem because it doesn't even remotely solve it or give the appearance of solving it, if anything it's guaranteed to make things go slower

Google did this a long time ago. It’s more nuanced than you think. Two requests would be issued and the first acknowledged one cause the second to be cancelled. There was a public paper on this from Dean iirc and the method is a decade old.

The technique is called "request hedging".

Re: Twitter Is DDOSing Itself

#710
post #645

Earlier quoted context omitted.

No, look closely: the animation does not show the console, it shows the network pane. Each line is a (presumably unsuccessful) request to the Twitter backend. If a lot of people have the Twitter web frontend open and running in this state, it could in fact overload the backend.

Sure, I mis-spoke in saying it was console.log spam, that’s not specifically what’s happening. But let’s be real, if you’re getting a 429 response that is at most a cached hash table lookup per request, for a site has probably top 50 in the world in terms of “scaling to handle heavy request loads” for a decade. There’s just no chance this specific thing is actually what’s causing issues for Twitter, it’s obviously a…

Agreed, it's fud - and there are professional botnets on HackerNews to amplify what they want.

What I've noticed is there's little technical discussion, and more so emotional-provoking replies.

Post reply on HN