Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The impact of Prince’s death on Wikipedia

blog.wikimedia.org

Post reply on HN
Print view View original

41–50 of 112 posts

Re: The impact of Prince’s death on Wikipedia

#41

Post by sparkzilla »

yeukhon wrote: post #26

They mentioned 5M views within 24 hours of Michael Jackson's death. With over 3B Internet users out there, I am actually a little surprised how small the spike was. Did they only count English Wikipedia? Even so I am quite surprised. I would expect 10-20M at least. Similarly, many young people like myself have never heard of Prince, I had to look him up to find out who he truly was.

Just out of interest, did you go directly to Wikipedia to find the information, or did you go to Google, which then led you to Wikipedia?

Re: The impact of Prince’s death on Wikipedia

#43

Post by brianwawok »

danielrhodes wrote: post #40
Washuu wrote: post #36

Earlier quoted context omitted.

Varnish does handle the mass of logged out cached requests. However, because they were having such a high amount of page edits per second the cache in Varnish would only by valid for about a second. Then a flood of logged out users hit the servers at the same requesting the uncached page to be rendered. The PoolCounter extensions keeps the web servers under control and by throttling requests for page rendering.

Correct me if I am wrong, but I thought Varnish has support for limiting concurrent backend fetches to the same resource.

Across a cluster of varnishes? I think that limit is per varnish.

Re: The impact of Prince’s death on Wikipedia

#44

Post by duskwuff »

erelde wrote: post #22
Buge wrote: post #6

Interesting how in the graph it looks like some people found out about 25 minutes before it was more publicly found out.

I'm seeing an exponential curve, isn't it what we should expect ?

Look very closely at the section of the graph starting around 4:20 PM. There's a small but significant increase in hits before the big spike starts around 4:50.

It's easier to see on the full resolution graph:

https://upload.wikimedia.org/wikipedia/commons/f/f2/Prince_a...

Re: The impact of Prince’s death on Wikipedia

#46

Post by danielrhodes »

brianwawok wrote: post #43
danielrhodes wrote: post #40

Earlier quoted context omitted.

Correct me if I am wrong, but I thought Varnish has support for limiting concurrent backend fetches to the same resource.

Across a cluster of varnishes? I think that limit is per varnish.

No, definitely not across a cluster (although that would be quite nifty). Even on a single node that would reduce the thundering herd effect substantially.

Re: The impact of Prince’s death on Wikipedia

#47

Post by atdt »

semi-extrinsic wrote: post #9

For others who were left scratching their heads at what exactly this pop-sci-explained PoolCounter mechanism actually is: https://wikitech.wikimedia.org/wiki/PoolCounter TL;DR: It's a limiter on how many workers start rendering the new page version when the old page version in cache has been invalidated.

Yeah, I am the engineer mentioned in the article, and I agree the explanation doesn't really work. The pieces of the explanation that ended up in the article itself don't add up to a coherent explanation. The fault for that is mostly mine. In hindsight, my original explanation was too long and too elaborate to be helpful. It's a good reminder that it is easy to go to far with an analogy and end up complicating the thing you were trying to simplify. Oh well, live and learn :)

There are more (coherent and to-the-point) details about PoolCounter in the prologue to PoolCounter.php in MediaWiki's source tree:

https://github.com/wikimedia/mediawiki/blob/1617e7822eaf7426...

And in a short blog post by Domas Mituzas, who is the original author of PoolCounter:

https://dom.as/2009/06/26/embarrassment/

Re: The impact of Prince’s death on Wikipedia

#48

Post by fleitz »

nothrabannosir wrote: post #16
zappo2938 wrote: post #11

Earlier quoted context omitted.

Can't caching a page with varnish and memcache handle this?

They cache the bejeezus out of their pages. Problems come up when a lot of people want to edit a page, an inherently uncacheable operation.

It's possible to use stacks to 'cache' writes in scenarios like this.

Writes to the same object go in the same stack, iterate over stacks, pop the first item, write it, clear the stack.

It works miracles for ephemeral data like wikipedia edits.

If you have extremely spikey load on servers, stacks are also a great replacement for queues, admit that during the deluge some portion of queries will timeout and go unanswered, instead of trying to process queries that are likely to timeout, simply process the first query on the stack and don't waste time processing the ones bound to fail.

Re: The impact of Prince’s death on Wikipedia

#49

Post by aaron695 »

JBReefer wrote: post #3

This is so impressive, to see behind the curtains of what has become the central repository of humanities knowledge, during a moment of loss of one of humanity's greats.

> during a moment of loss of one of humanity's greats.

Compared to, lets say Bill Gates who's saved millions of lives?

Even artistically, I'm not sure Prince was up there in the top 1%

The power of marketing.....

Re: The impact of Prince’s death on Wikipedia

#50

Post by homero »

atdt wrote: post #47
semi-extrinsic wrote: post #9

For others who were left scratching their heads at what exactly this pop-sci-explained PoolCounter mechanism actually is: https://wikitech.wikimedia.org/wiki/PoolCounter TL;DR: It's a limiter on how many workers start rendering the new page version when the old page version in cache has been invalidated.

Yeah, I am the engineer mentioned in the article, and I agree the explanation doesn't really work. The pieces of the explanation that ended up in the article itself don't add up to a coherent explanation. The fault for that is mostly mine. In hindsight, my original explanation was too long and too elaborate to be helpful. It's a good reminder that it is easy to go to far with an analogy and end up complicating the th…

Basically a page view checks the cache and rebuilds if necessary. Thousands of page hits in the same second before the build is over starts thousands of parallel rebuilds.
Post reply on HN