Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The impact of Prince’s death on Wikipedia

blog.wikimedia.org

Post reply on HN
Print view View original

101–110 of 112 posts

Re: The impact of Prince’s death on Wikipedia

#101

Post by DanWaterworth »

hedgehog wrote: post #85
atdt wrote: post #47

Earlier quoted context omitted.

Yeah, I am the engineer mentioned in the article, and I agree the explanation doesn't really work. The pieces of the explanation that ended up in the article itself don't add up to a coherent explanation. The fault for that is mostly mine. In hindsight, my original explanation was too long and too elaborate to be helpful. It's a good reminder that it is easy to go to far with an analogy and end up complicating the th…

There is a stochastic approach that can be adapted to address this problem, I think I first saw it at IMVU in 2009 but conveniently Wikipedia has a good reference now: https://en.wikipedia.org/wiki/Cache_stampede#Probabilistic_e... The advantage is less coordination is necessary and you should be able to get down to a single concurrent rerender per page.

Wow, that's a nice technique, but in this case, isn't the rendered page invalidated by changes to the source, rather than over time?

I suppose, in this case you could use the time since invalidation as your input. The downside is that changes in the source aren't immediately reflected in the rendered output, especially for infrequently updated pages.

Re: The impact of Prince’s death on Wikipedia

#102

Post by andrewingram »

homero wrote: post #70
taf2 wrote: post #67

Earlier quoted context omitted.

Also known as the thundering herd...

Yeah I never really solved it while I was using memcached but I'm not Wikipedia

The way i've seen it done is something like this:

* Cache never automatically expires, but you do have some notion of staleness * Whenever you request data, you get it from the cache (if it exists), and check for staleness, so you cached data needs to know when it was cached. * Return the data as usual, but if the data was also stale, you fire off a worker to update the cached data. * If you have lots of requests happening at the same time, you have a system for seeing if a worker already exists, to ensure that you only create one (for each piece of cached data). * For the time it takes for the worker to complete, you have to be okay with serving stale data, in most cases this is okay.

There's an edge case missed here, which is what to do when the cache is empty (either because it's one of the first requests, or because the cached data has been evicted). That's up to you, depending on your use case. You can basically either return a default value, you can pre-warm your cache, or you can let the requests hang until the data is ready.

Re: The impact of Prince’s death on Wikipedia

#103

Post by homero »

forgotpwtomain wrote: post #86
homero wrote: post #50

Earlier quoted context omitted.

Basically a page view checks the cache and rebuilds if necessary. Thousands of page hits in the same second before the build is over starts thousands of parallel rebuilds.

Why isn't it a possibility to rebuild and invalidate the cache only when the rebuild is finished?

Because it's usually triggered with the cache expiring then it's not available and every page view attempts rebuilds. Otherwise the cache and time will be separate. I store the cache in memcached with an expiration to detect it expiring

Re: The impact of Prince’s death on Wikipedia

#104

Post by homero »

andrewingram wrote: post #102
homero wrote: post #70

Earlier quoted context omitted.

Yeah I never really solved it while I was using memcached but I'm not Wikipedia

The way i've seen it done is something like this: * Cache never automatically expires, but you do have some notion of staleness * Whenever you request data, you get it from the cache (if it exists), and check for staleness, so you cached data needs to know when it was cached. * Return the data as usual, but if the data was also stale, you fire off a worker to update the cached data. * If you have lots of requests hap…

I never looked into whether memcached will give you the creation date, I just let memcached expire it itself

Re: The impact of Prince’s death on Wikipedia

#105

Post by yeukhon »

sparkzilla wrote: post #41
yeukhon wrote: post #26

They mentioned 5M views within 24 hours of Michael Jackson's death. With over 3B Internet users out there, I am actually a little surprised how small the spike was. Did they only count English Wikipedia? Even so I am quite surprised. I would expect 10-20M at least. Similarly, many young people like myself have never heard of Prince, I had to look him up to find out who he truly was.

Just out of interest, did you go directly to Wikipedia to find the information, or did you go to Google, which then led you to Wikipedia?

Always Google, which leads me to Wikipedia. Almost literally every time...

Re: The impact of Prince’s death on Wikipedia

#106

Post by yeukhon »

cooper12 wrote: post #73
yeukhon wrote: post #26

They mentioned 5M views within 24 hours of Michael Jackson's death. With over 3B Internet users out there, I am actually a little surprised how small the spike was. Did they only count English Wikipedia? Even so I am quite surprised. I would expect 10-20M at least. Similarly, many young people like myself have never heard of Prince, I had to look him up to find out who he truly was.

They recently overhauled it, [0] but back then the pageviews [1] wouldn't count mobile users. You can see the old stats for Jackon's page here: http://stats.grok.se/en/200906/Michael%20Jackson . The actual number is 5,875,404 views within that day (in whichever time zone) and is for the English version of the article specifically. [0]: https://blog.wikimedia.org/2015/12/14/pageview-data-easily-a... [1]: https://en.wi…

Thanks. Yeah, I think mobile viewer would be a substainal amount, but desktop user count is still below my personal expectation. We are talking about 850M English speaking Internet users :( only 6M page view within 24 hours is really quite low.

Re: The impact of Prince’s death on Wikipedia

#107

Post by andrewingram »

andrewingram wrote: post #102
homero wrote: post #70

Earlier quoted context omitted.

Yeah I never really solved it while I was using memcached but I'm not Wikipedia

The way i've seen it done is something like this: * Cache never automatically expires, but you do have some notion of staleness * Whenever you request data, you get it from the cache (if it exists), and check for staleness, so you cached data needs to know when it was cached. * Return the data as usual, but if the data was also stale, you fire off a worker to update the cached data. * If you have lots of requests hap…

I should've fixed the formatting whilst it was still editable :(

A former colleague of mine built a Django implementation of this pattern, which is pretty useful: https://github.com/codeinthehole/django-cacheback

Re: The impact of Prince’s death on Wikipedia

#108

Post by quaristice »

michaelmrose wrote: post #77
quaristice wrote: post #57

Earlier quoted context omitted.

Well, I wasn't really responding to the Bill Gates part in particular. Gates was a bit of a megalomaniac with MS. But he's doing awesome stuff with the money now. As person I'm not aware of him ever being a jerk.

So if you gain lots of money via semi nefarious means what fraction do you have to dedicate to good works before the earlier wrong is cancelled? I know that he didn't gas 6 million people but letting people buy their way out of moral debt with a fraction of the money they gained still seems horribly repugnant.

First, what moral debt? Second, I imagine the sum total of his humanitarian efforts are greater than the total charity if all of those dollars remained in the pockets of each person who bought windows 95 et al. So repugnant seems like a real stretch.

Re: The impact of Prince’s death on Wikipedia

#109

Post by fleitz »

chris_wot wrote: post #68
fleitz wrote: post #48

Earlier quoted context omitted.

It's possible to use stacks to 'cache' writes in scenarios like this. Writes to the same object go in the same stack, iterate over stacks, pop the first item, write it, clear the stack. It works miracles for ephemeral data like wikipedia edits. If you have extremely spikey load on servers, stacks are also a great replacement for queues, admit that during the deluge some portion of queries will timeout and go unanswer…

Doesn't caching writes from multiple different servers potentially cause consistency and durability concerns? I know MongoDB still haven't marked the bug [1] reported by Kyle Kingsbury [2] that found stale reads on all consistency and write concern levels... 1. https://jira.mongodb.org/plugins/servlet/mobile#issue/SERVER... 2. https://aphyr.com/posts/322-jepsen-mongodb-stale-reads

Yes, but this is wikipedia, the entire premise of it is eventual consistency. The idea being that the most recent update to a page is the correct one.

Re: The impact of Prince’s death on Wikipedia

#110

Post by forgotpwtomain »

homero wrote: post #103
forgotpwtomain wrote: post #86

Earlier quoted context omitted.

Why isn't it a possibility to rebuild and invalidate the cache only when the rebuild is finished?

Because it's usually triggered with the cache expiring then it's not available and every page view attempts rebuilds. Otherwise the cache and time will be separate. I store the cache in memcached with an expiration to detect it expiring

You could queue an async job to rebuild what was stored in the cache right before the expiration time when you update / renew the expiration on the cached item?
Post reply on HN