Given how much simpler it is to count total views than unique user views, why is it more valuable to count unique user views?
Because it's a metric advertisers care about.
View Counting at Reddit
91–100 of 121 posts
Re: View Counting at Reddit
#92How do you test a system like this for accuracy? Is this done by simulating millions of unique requests?
The algorithm's accuracy is known. From the wiki[1]: The HyperLogLog algorithm is able to estimate cardinalities of > 10^9 with a typical error rate of 2% [1] https://en.wikipedia.org/wiki/HyperLogLog
But what about the implementation accuracy? :)
Re: View Counting at Reddit
#93Probably noob question, but:
>> Nazar will then alter the event, adding a Boolean flag indicating whether or not it should be counted, before sending the event back to Kafka.
Why don't they just discard it instead of reputting the event back to Kafka?
Re: View Counting at Reddit
#94"We want to better communicate the scale of Reddit to our users." If that's true why did they hide vote numbers on comments and posts? It used to say "xxx upvotes xxx downvotes" now it just gives a number and hides that.
Suppose there's a comment with 12 upvotes and 207 downvotes. Now suppose you're reddit and you want to make this comment seem more popular than it actually is. You could slowly remove the downvotes, but attentive people refreshing the page will see the number shrinking and become suspicious, because real users never suddenly do a mass retraction of their votes. You could slowly add a bunch of upvotes, but then people…
Except that vote fuzzing was always a thing, so when you'd see 1000 upvotes and 100 downvotes, that could be off by an enormous amount (I think a reddit admin said by a factor of 5x or 10x in extreme cases, ie. frontpage posts).
Re: View Counting at Reddit
#95Earlier quoted context omitted.
Why can't they just associate a list of viewed posts with each user, or list of users that viewed a post with each post, and check that? I don't get why this needs any consideration.
They addressed your second point in the article. On a popular post, you would be storing several megabytes of data to capture/relate each unique user that visited. That gets expensive at scale. HLL takes then down to a few kilobytes, less than 1% of the original size. For your first suggestion, you would have to do a very expensive look up. You couldn't cache it effectively due to the requirement of near real time s…
I've had a "phases of computing" article percolating for a while to this end. Problems aren't just harder at scale, but they actively change their observable properties because of the stressors involved and where they crop up.
Re: View Counting at Reddit
#96Earlier quoted context omitted.
Actually, reddit has that. You can save threads or comments, and sort by latest.
And with reddit gold you can even have it hilight all new replies regardless of sort order. This can be really useful for viewing older discussions and see what's been added.
The RES browser extension does this without needing gold, or am I wrong about that?
Re: View Counting at Reddit
#97Earlier quoted context omitted.
Can't you just delay updating the count by some random number of minutes/hours?
That be easy to test though if you were bot was effective or not, just post to unpopular subreddits, make bot votes on those submissions, then check back the next day. If votes not counted, then your bot is being ignored and you'd move on to changing your IP address or building your next bot or such.
There's no reason this exact same method won't work given their current practice though.
Re: View Counting at Reddit
#98Wouldn't it had been easier to simply increment a counter for each visit and then set a short lived cookie in the browser for that post? And put the spam detection system before the counter increment
I think they said the product decision was made primarily to prevent abuse. I don't think a cookie could stop a sophisticated abuser.
[deleted]
Re: View Counting at Reddit
#99Earlier quoted context omitted.
How about, here and on reddit, being able to mark threads you're interested in, and having a page where you can see those sorted by last reply. On HN, make that page refresh every 15 or 60 minutes or whatever. Heck, once every 24 hours would be enough... sometimes I just want to talk about the things that interest me, with the people that are interested in them. I would love to be able to think on something for a few…
> As they are now, reddit and HN are require you to be there when something is "current", and that's ultimately not that much better than TV. One thing I don't think people have addressed is that that's the goal for them; it's not necessarily to be "useful" to the end user. If they can make users feel like they have to be on reddit 24/7 to stay "current" and not miss anything, then they will. Facebook does the same t…
For Reddit, this is understandable. For HN, an aggregator which tries to take the high road (or claims to), it is not.
Re: View Counting at Reddit
#100Earlier quoted context omitted.
The algorithm's accuracy is known. From the wiki[1]: The HyperLogLog algorithm is able to estimate cardinalities of > 10^9 with a typical error rate of 2% [1] https://en.wikipedia.org/wiki/HyperLogLog
But what about the implementation accuracy? :)
Tests against both historical and synthetic datasets.