Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Built r/Place

redditblog.com

Post reply on HN
Print view View original

151–160 of 255 posts

Re: How We Built r/Place

#151

Post by raziel2p »

vmasto wrote: post #6

> Users can place one tile every 5 minutes, so we must support an average update rate of 100,000 tiles per 5 minutes (333 updates/s). It only takes a couple of outliers to bring everything down. I'm not exactly well-versed in defining specs for large scale backend apps (not a back-end engineer) but it seems to me that preparing for the average would not be a wise decision? For example, designing with an average of a…

With 100K users and a minimum cooldown time of 5 minutes, there's 100K tiles per 5 minutes would be an upper limit. Only bots would hit that 5 minute cooldown every single time, and bots make up a very small minority of the users. They even point out later in the article that they only peaked at 200 updates per second.

> For example, designing with an average of a million requests per day in mind would probably fail, since you get most of that traffic during daytime and far more less at the nightly hours.

That depends. If you're talking about a public website, then yes. But this is just a single API endpoint with a very fixed time-based rate limit.

Re: How We Built r/Place

#152

Post by gooeyblob »

77pt77 wrote: post #3
mozumder wrote: post #2

> We actually had a race condition here that allowed users to place multiple tiles at once. There was no locking around the steps 1-3 so simultaneous tile draw attempts could all pass the check at step 1 and then draw multiple tiles at step 2. This is why you use a proper database. I'd probably add a Postgres table to record all user activity, and use that to lock out users for 5 minutes as an initial filter. Have tr…

These no-SQL solutions seem shoehorned many times. Like people decide to use them from the get-go and then come up with a justification.

We've been using Cassandra for 7 years since 0.7.

Re: How We Built r/Place

#153

Post by [deleted] »

ChicagoBoy11 wrote: post #124

I love write-ups like this because they are such a nice contrast to the too-common comments on Reddit and HN where people claim that they could rebuild FB or Uber as a side project. Something as superficially trivial as the r/Place requires a tremendous amount of effort to run smoothly and there are countless gotchas and issues that you'd never even begin to consider unless you really tried to implement it yourself.…

[deleted]

Re: How We Built r/Place

#154

Post by kzrdude »

pitaj wrote: post #23

r/Place is really awesome. This is how you grow the community. The 2D and 3D timelapses are super cool to watch, as well. Glad Reddit decided to make this a full-time thing.

Full time? I haven't heard that anywhere.

Re: How We Built r/Place

#155

Post by _m8fo »

samfisher83 wrote: post #120
seankimdesign wrote: post #75

What a fantastic writeup. I had some vague ideas regarding the challenges involved to build an application of such scale, but the article really makes it clear for everyone the amount of decision points encountered as well as why certain solutions were selected. I also like the way the article is broken down into the backend, API, frontend and mobile. This isolated approach really highlights the different struggles e…

That writeup may have taken more time than building it out. It seems quite thorough. Edit: why am I getting down voted for praising a write-up?

Because despite being thorough, there's no way this write-up could be completed slower than building the actual thing they're describing.

Re: How We Built r/Place

#156

Post by kzrdude »

maaaats wrote: post #19

Interesting how big the Norwegian and Swedish flags got, given our small populations.

Populations are not so small in reddit terms. Sweden probably has an equal population to Italy or even more, on reddit.

Re: How We Built r/Place

#157

Post by petercooper »

eblanshey wrote: post #115
seankimdesign wrote: post #75

What a fantastic writeup. I had some vague ideas regarding the challenges involved to build an application of such scale, but the article really makes it clear for everyone the amount of decision points encountered as well as why certain solutions were selected. I also like the way the article is broken down into the backend, API, frontend and mobile. This isolated approach really highlights the different struggles e…

Agreed, great write-up. Anyone have other recommended links to write-ups about specific challenges and how to make them scale?

Here are some recent ones that have been popular:

http://highscalability.com/blog/2016/4/20/how-twitter-handle...

https://redditblog.com/2017/1/17/caching-at-reddit/

https://blog.twitter.com/2017/the-infrastructure-behind-twit...

https://www.netlify.com/blog/2017/03/16/smashing-magazine-ju...

https://nickcraver.com/blog/2016/02/03/stack-overflow-a-tech...

https://engineering.linkedin.com/blog/2016/10/instant-messag...

http://techblog.netflix.com/2016/08/building-fastcom.html

http://highscalability.com/blog/2016/9/28/how-uber-manages-a...

Or a recent writeup on a challenge building IMDB's forums in 2001(!) https://www.beatworm.co.uk/blog/internet/imdb-boards-no-more .. fun and scary in equal measures.

(I'm plucking these from things were popular recently in our http://webopsweekly.com/ :-))

Re: How We Built r/Place

#158

Post by Macha »

celticninja wrote: post #127
ClassyJacket wrote: post #125

Earlier quoted context omitted.

The search is consistently broken, but the rest of the site seems to have good uptime now, much better than it once was.

Search has never worked. I always use Google site: modifier and have never had an issue.

Search worked pretty reliably up until 7-8 months ago for me

Re: How We Built r/Place

#159

Post by dkasper »

biot wrote: post #146

> At the peak of r/place the websocket service was > transmitting over 4 gbps (150 Mbps per instance > and 24 instances). What does Reddit use for serving up this much websocket traffic? Something open source, or is it custom built?

It's open source and custom built. https://github.com/reddit/reddit-service-websockets

Re: How We Built r/Place

#160

Post by Ono-Sendai »

wyager wrote: post #80

Why use Redis and multiple machines instead of keeping it in RAM on a single machine? I'm not claiming the Reddit people did anything wrong; they have a lot more experience than me here obviously. I'm just trying to figure out why they couldn't do something simpler. 333 updates/sec to a 500kB packed array, coupled with cooldown logic, should have a negligible performance cost and can easily be done on a single thread…

Updating a bitmap 333 times a second is trivial with one core. Handling 100K websocket connections is the tricky bit I think!
Post reply on HN