31–40 of 112 posts
For others who were left scratching their heads at what exactly this pop-sci-explained PoolCounter mechanism actually is: https://wikitech.wikimedia.org/wiki/PoolCounter TL;DR: It's a limiter on how many workers start rendering the new page version when the old page version in cache has been invalidated.
+1 for whoever named it the "Michael Jackson problem". I'm going to start using that if I ever encounter the same issue myself.
"He was ... known for, among many other things, ... a performance at Super Bowl XLI in a raining downpour in front of over a hundred million people." Typo and/or I call bullshit.
This[1] seems to indicate viewership of the halftime show peaked at 140M. 1: https://web.archive.org/web/20090412054158/http://www.suntim...
For others who were left scratching their heads at what exactly this pop-sci-explained PoolCounter mechanism actually is: https://wikitech.wikimedia.org/wiki/PoolCounter TL;DR: It's a limiter on how many workers start rendering the new page version when the old page version in cache has been invalidated.
+1 for whoever named it the "Michael Jackson problem". I'm going to start using that if I ever encounter the same issue myself.
Earlier quoted context omitted.
This[1] seems to indicate viewership of the halftime show peaked at 140M. 1: https://web.archive.org/web/20090412054158/http://www.suntim...
Okay, I can see where you're coming from but the sentence was "raining downpour in front of over a hundred million people." How does that NOT imply a live audience of a hundred million? I can't believe people are downvoting me for this. The sentence is absurd as written.
Earlier quoted context omitted.
This[1] seems to indicate viewership of the halftime show peaked at 140M. 1: https://web.archive.org/web/20090412054158/http://www.suntim...
Okay, I can see where you're coming from but the sentence was "raining downpour in front of over a hundred million people." How does that NOT imply a live audience of a hundred million? I can't believe people are downvoting me for this. The sentence is absurd as written.
I don't think WMF staff are credited enough for the work they do in keeping Wikipedia running. They seriously know how to scale, I think the only ones better than them are honestly Facebook and Twitter!
Can't caching a page with varnish and memcache handle this?
Even in peak it's just ~800 hits per second - it shows how is irrelevant the C10k problem (yes, I know it's not exactly about hits per second, but still).
(1) 800rps TO THAT PAGE is the metric. The entire rest of Wikipedia was still getting traffic, and as an educated guess I would estimate raw traffic to be on the order of magnitude of 3-4krps (across editing and views). They are quite open with operations and if I weren't mobile I could probably find the accurate answer.
(2) There are much higher traffic properties. I'm aware of one property beyond 200krps in aggregate.
If you had said "most people don't have to worry about C10k," then I'm on board. That's true. Irrelevant? Far from it.
And yes, query rate and connection count have a complicated relationship. You need three or four other metrics to explain their relationship, but raw query rate is a good yardstick for active connections when combined with quantiled request latency. (Not averaged.) Simple example: a 750ms 95th% page hit 10,000 times per second is almost certainly far > C10k because of the outliers.
Now I will grant that C10k itself is somewhat irrelevant, yes, but not for the reason you are saying. It was defined in an age when 10,000 active connections was pretty surprising (didn't it come from FTP or some other heavy eyeball protocol?). These days with long poll apps, long-running protocols, and so on, millions of open connections are quite common at consumer scale. I find C1M far more interesting these days. C10M is still kinda nuts, but does exist in the magical world of metal and fiber and hot aisles and all those great things that nobody uses anymore (depressingly).
Earlier quoted context omitted.
Okay, I can see where you're coming from but the sentence was "raining downpour in front of over a hundred million people." How does that NOT imply a live audience of a hundred million? I can't believe people are downvoting me for this. The sentence is absurd as written.
Downvotes aren't punishment. Up/down votes indicate "this comment ought to be more/less prominent on the page". The sentence was somewhat ambiguous and your misunderstanding is understandable. But your comment is not useful to the discussion because most people understood and in any case it's tangential.
Earlier quoted context omitted.
2m per second ? These guys must have squeezed every ounce of performance out of Beam.
2m refers to concurrent connections, not rate.
Earlier quoted context omitted.
Can't caching a page with varnish and memcache handle this?
Varnish does handle the mass of logged out cached requests. However, because they were having such a high amount of page edits per second the cache in Varnish would only by valid for about a second. Then a flood of logged out users hit the servers at the same requesting the uncached page to be rendered. The PoolCounter extensions keeps the web servers under control and by throttling requests for page rendering.