Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

View Counting at Reddit

redditblog.com

Post reply on HN
Print view View original

71–80 of 121 posts

Re: View Counting at Reddit

#71

Post by sethammons »

jimmaswell wrote: post #64
alzaeem wrote: post #48

So how do they determine whether a user has viewed a post already? I would think that unique counting is accomplished using the hyperloglog counter, but the article says that this decision is made by the Nazar system, which doesn't use the hyperloglog counter in Redis.

Why can't they just associate a list of viewed posts with each user, or list of users that viewed a post with each post, and check that? I don't get why this needs any consideration.

They addressed your second point in the article. On a popular post, you would be storing several megabytes of data to capture/relate each unique user that visited. That gets expensive at scale. HLL takes then down to a few kilobytes, less than 1% of the original size.

For your first suggestion, you would have to do a very expensive look up. You couldn't cache it effectively​ due to the requirement of near real time stats. You could improve look up time using columnar storage, but the performance and memory usage will be nowhere near as nice as with HLL.

Problems are harder at scale.

Re: View Counting at Reddit

#72

Post by heifer2822 »

tudorconstantin wrote: post #14

Wouldn't it had been easier to simply increment a counter for each visit and then set a short lived cookie in the browser for that post? And put the spam detection system before the counter increment

I think they said the product decision was made primarily to prevent abuse. I don't think a cookie could stop a sophisticated abuser.

Re: View Counting at Reddit

#73

Post by gniv »

tudorconstantin wrote: post #14

Wouldn't it had been easier to simply increment a counter for each visit and then set a short lived cookie in the browser for that post? And put the spam detection system before the counter increment

That might work, but the cookie would be huge for people who read a lot of reddit threads, no?

Re: View Counting at Reddit

#74

Post by sethammons »

diziet wrote: post #62
tudorconstantin wrote: post #14

Wouldn't it had been easier to simply increment a counter for each visit and then set a short lived cookie in the browser for that post? And put the spam detection system before the counter increment

Yes, that was my idea as well. They must have some sort of cache system for serving basic user meta-data at scale when a page is loaded, and they could add a time-expiring list post ids of the posts viewed by a user to do detection on a per user basis on the backend. I think they want to break it into different services for (whatever) reasons. Running a counter across a sharded in memory cache implementation (like Re…

I'm not following your solution for how it handles the unique visitor counting. You have to know when to increment the counter. Thus why they used the HLL.

Re: View Counting at Reddit

#76

Post by joombaga »

sanswork wrote: post #38
wordupmaking wrote: post #27

Earlier quoted context omitted.

But that doesn't "prevent spambots etc.", that merely prevents people from easily and instantly figuring out whether their bots are detected yet. It doesn't stop them from spamming votes, or from making their bots more elaborate regardless of whether they have been detected yet. I don't know all the motivations for spam bots or how people who make them tick, but I'd figure for a significant number the crucial bit is…

The point isn't to prevent them which is near on impossible but to get them to waste enough resources on non-visible actions regularly enough that it isn't economically viable to continue trying to spam the site.

Even so, I think GP's point is still valid in that it doesn't achieve that desired effect, but of course Reddit has those stats and I do not.

Re: View Counting at Reddit

#78

Post by jliptzin »

Klathmon wrote: post #70
ma2rten wrote: post #69

Earlier quoted context omitted.

I don't quite see the connection. How exactly does this deter bots?

It's difficult to see if their votes are counting, allowing Reddit to silently-ignore their votes without them knowing.

Can't you just delay updating the count by some random number of minutes/hours?

Re: View Counting at Reddit

#79

Post by antirez »

bognition wrote: post #57
BillinghamJ wrote: post #36

Earlier quoted context omitted.

Redis writes are atomic - you just use the increment function

Writes are atomic in redis because redis is single threaded. So you are bounded by how fast redis can write. If you try to write any faster then redis can handle you'll get queueing or errors.

The wonderful thing about HyperLogLogs is that you can split the counter in N servers and "merge" the registers later, in case you want an architecture that shards the same counter in multiple servers. But sharding directly by resource looks simpler actually...

Re: View Counting at Reddit

#80

Post by nowarninglabel »

jliptzin wrote: post #78
Klathmon wrote: post #70

Earlier quoted context omitted.

It's difficult to see if their votes are counting, allowing Reddit to silently-ignore their votes without them knowing.

Can't you just delay updating the count by some random number of minutes/hours?

That be easy to test though if you were bot was effective or not, just post to unpopular subreddits, make bot votes on those submissions, then check back the next day. If votes not counted, then your bot is being ignored and you'd move on to changing your IP address or building your next bot or such.
Post reply on HN