Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Built r/Place

redditblog.com

Post reply on HN
Print view View original

11–20 of 255 posts

Re: How We Built r/Place

#11

Post by d23 »

mozumder wrote: post #2

> We actually had a race condition here that allowed users to place multiple tiles at once. There was no locking around the steps 1-3 so simultaneous tile draw attempts could all pass the check at step 1 and then draw multiple tiles at step 2. This is why you use a proper database. I'd probably add a Postgres table to record all user activity, and use that to lock out users for 5 minutes as an initial filter. Have tr…

So in that case, each pixel would be stored as a separate row in a relational database? And to query the whole canvas you'd query a million rows on every read?

I lean towards just using the ratelimiting stuff we already have in place (via memcached, which we talked about in a previous post). We just overlooked it.

Re: How We Built r/Place

#12

Post by fidget »

bsimpson63 wrote: post #10
mozumder wrote: post #2

> We actually had a race condition here that allowed users to place multiple tiles at once. There was no locking around the steps 1-3 so simultaneous tile draw attempts could all pass the check at step 1 and then draw multiple tiles at step 2. This is why you use a proper database. I'd probably add a Postgres table to record all user activity, and use that to lock out users for 5 minutes as an initial filter. Have tr…

At reddit it's much easier for us to stand up a new Cassandra column family than a new postgres table (not saying this is how it should be, but just how it is). All we needed to do here was add some simple locking and we would have been fine.

Are you suggesting that you had a concurrency bug that was solvable without changing your entire storage layer? Heresy..

Re: How We Built r/Place

#13

Post by [deleted] »

vmasto wrote: post #6

> Users can place one tile every 5 minutes, so we must support an average update rate of 100,000 tiles per 5 minutes (333 updates/s). It only takes a couple of outliers to bring everything down. I'm not exactly well-versed in defining specs for large scale backend apps (not a back-end engineer) but it seems to me that preparing for the average would not be a wise decision? For example, designing with an average of a…

[deleted]

Re: How We Built r/Place

#14

Post by andoon »

vmasto wrote: post #6

> Users can place one tile every 5 minutes, so we must support an average update rate of 100,000 tiles per 5 minutes (333 updates/s). It only takes a couple of outliers to bring everything down. I'm not exactly well-versed in defining specs for large scale backend apps (not a back-end engineer) but it seems to me that preparing for the average would not be a wise decision? For example, designing with an average of a…

The entire reddit website goes down every night, especially during weekends, sport matches, etc, so there you have your answer.

Re: How We Built r/Place

#15

Post by bsimpson63 »

vmasto wrote: post #6

> Users can place one tile every 5 minutes, so we must support an average update rate of 100,000 tiles per 5 minutes (333 updates/s). It only takes a couple of outliers to bring everything down. I'm not exactly well-versed in defining specs for large scale backend apps (not a back-end engineer) but it seems to me that preparing for the average would not be a wise decision? For example, designing with an average of a…

This was just a back of the napkin estimation. I think at one point we calculated that we'd be able to support one tile placement per user per second.

Re: How We Built r/Place

#17

Post by [deleted] »

vmasto wrote: post #6

> Users can place one tile every 5 minutes, so we must support an average update rate of 100,000 tiles per 5 minutes (333 updates/s). It only takes a couple of outliers to bring everything down. I'm not exactly well-versed in defining specs for large scale backend apps (not a back-end engineer) but it seems to me that preparing for the average would not be a wise decision? For example, designing with an average of a…

[deleted]

Re: How We Built r/Place

#18

Post by celticninja »

andoon wrote: post #14
vmasto wrote: post #6

> Users can place one tile every 5 minutes, so we must support an average update rate of 100,000 tiles per 5 minutes (333 updates/s). It only takes a couple of outliers to bring everything down. I'm not exactly well-versed in defining specs for large scale backend apps (not a back-end engineer) but it seems to me that preparing for the average would not be a wise decision? For example, designing with an average of a…

The entire reddit website goes down every night, especially during weekends, sport matches, etc, so there you have your answer.

Is that hyperbole or are you really experiencing that much downtime of Reddit? I see it occasionally but it's never down for long, the odd "servers are busy" message usually disappears after a single refresh.

Re: How We Built r/Place

#20

Post by tyrust »

vmasto wrote: post #6

> Users can place one tile every 5 minutes, so we must support an average update rate of 100,000 tiles per 5 minutes (333 updates/s). It only takes a couple of outliers to bring everything down. I'm not exactly well-versed in defining specs for large scale backend apps (not a back-end engineer) but it seems to me that preparing for the average would not be a wise decision? For example, designing with an average of a…

They might have phrased this poorly.

>We should support at least 100,000 simultaneous users.

This line makes me think that this is what they expected the peak (or near peak) to be.

>Users can place one tile every 5 minutes, so we must support an average update rate of 100,000 tiles per 5 minutes (333 updates/s).

So assuming that they mean that 100k is the peak and that clients are limited to 1 update per 5 minutes, they can expect 333 updates per second on average. The "average" is taken over this 5 minute period. This average represents the number of queries per second they will get if everyone's 5 minute cooldown is spread out evenly over each 5 minute period.

It is possible, for example, for half of the peak population's cooldown to expire at 1300 and the other half to expire at 1305. In this case the average updates/s over the 5 minute period from 1300-1305 would still be 333 updates/s even though there were really 2 bursts of 50k a second at 1300 and 1305. It's far more likely that cooldowns are not excessively stacked in this way, so you prepare for the average and hope for the best.

Post reply on HN