This is all great, but your search never works, still. It has been like that forever.
How We Built r/Place
111–120 of 255 posts
Re: How We Built r/Place
#112This reminds me of http://www.milliondollarhomepage.com/
Re: How We Built r/Place
#113Why use Redis and multiple machines instead of keeping it in RAM on a single machine? I'm not claiming the Reddit people did anything wrong; they have a lot more experience than me here obviously. I'm just trying to figure out why they couldn't do something simpler. 333 updates/sec to a 500kB packed array, coupled with cooldown logic, should have a negligible performance cost and can easily be done on a single thread…
> Why use Redis and multiple machines instead of keeping it in RAM on a single machine? Because machines go down. If you don't expect your hardware to fail at the most inopportune time, you'll be screwed when (not if) it happens.
I agree that machines go down, but there are sane (and safe!) ways to build this sort of thing without adding in cassandra and Redis. Additionally, the max placement rate of 333/s is reaaaaally slow! Maybe that's due to the websocket frontends, not the DB, but, that doesn't mean that's the most obvious way to build it.
The crux of the problem is that they need to mutate a relatively tiny amount of memory and have a rolling log of events for which only the last 5 minutes needs fast access. Also, if you can put all your state on one machine its far less likely that the one machine will die, than it is that at least one will die in a cluster of machines. Given the nature of the problem keeping all state on one machine seems pretty rational to me, so long as you have the ability to switch to a hot spare within a few minutes or so.
If I were to architect this for speed I would have two tiers: a websocket tier, and secondly a 'database' tier. The database would be a custom program that would:
0. Provide a simple Websocket API that would receive a write request and return either success if the user's write timer allowed it to write or failure if it didn't. This would also broadcast the state + deltas. 1. Keep the image in memory as a bitmap 2. Use rocksdb for tracking last user writes to enforce the 5m constraint. You could use an in memory map, but the nice thing about rocksdb is that it shouldn't blow up your heap. 3. Periodically flush the bitmap out to disk to timestamped files for snapshots 4. Keep the hashmap size small by evicting any keys past their time limit 5. Write rotating log files rotated every 5m or so to record the history of events for DR and also later analysis
Backing this sort of thing up is very simple. You just replicate the files using rsync or something like it. You may have some corruption on files that are partially written, but since we're opening and closing new files often you can choose how much data-loss you want to tolerate.
Restoration is as simple as re-reading the bitmap and reading the log files in reverse up to 5 minutes ago to see who still isn't allowed to write yet (thus reconstituting the hashmap). Let's remember, redis replication is async, so this has the same tradeoffs.
Re: How We Built r/Place
#114It's awesome project!
Re: How We Built r/Place
#115What a fantastic writeup. I had some vague ideas regarding the challenges involved to build an application of such scale, but the article really makes it clear for everyone the amount of decision points encountered as well as why certain solutions were selected. I also like the way the article is broken down into the backend, API, frontend and mobile. This isolated approach really highlights the different struggles e…
Agreed, great write-up. Anyone have other recommended links to write-ups about specific challenges and how to make them scale?
Re: How We Built r/Place
#116Earlier quoted context omitted.
I'm not the same guy you were just talking to but in my experience on mobile Safari, I get "something went wrong, visit the homepage" when I navigate around reddit far too often. What's funny is it actually tends to resolve itself if I wait a second or two, but yeah, reddit engineering and infrastructure is not well equipped to handle the amount of traffic they receive. It's gotten better, but it's still not what you…
The main website is OK as is i.reddit.com but their mobile client is hilariously bad in a "I can't believe they thought this was ready" way, it takes an age to load, it stalls out all the time, the tap targets are too small, it has the classic "oh you hit this when you meant that and then clicked that, lets spin the wheel on where you really end up" problem that slow mobile apps have. I know they deal with insane sca…
Last time I tried it, their app was even worse than the mobile website though.
Re: How We Built r/Place
#117This is awesome but man, reading the canvas portion was a bit distressing. I wonder why they didn't use a game engine to do this? All the work they did has been implemented already in several JS game engines, such as the one I help maintain (it's free and OSS), https://excaliburjs.com . We support all the features they needed including mobile & touch support. They could have also used Phaser ( http://phaser.io ) too…
Hi, I wrote the majority of that part of the project (canvas stuff) & that section of the article – the simple answer is that I have a lot of experience working with the canvas API directly, but little to no experience using any of the popular JS game engines out there (I played around with Phaser years ago, but not very much). I don't think it would've saved me any time to be honest.
Re: How We Built r/Place
#118Why use Redis and multiple machines instead of keeping it in RAM on a single machine? I'm not claiming the Reddit people did anything wrong; they have a lot more experience than me here obviously. I'm just trying to figure out why they couldn't do something simpler. 333 updates/sec to a 500kB packed array, coupled with cooldown logic, should have a negligible performance cost and can easily be done on a single thread…
> keeping it in RAM on a single machine > 500kB packed array, coupled with cooldown logic, should have a negligible performance cost and can easily be done on a single thread. That thread could interact via atomic channels with the CDN That's not simpler. We used tools that we're already using heavily in production and are comfortable with.
> That's not simpler.
With respect to your experience in the matter, I strongly disagree. What I described is complicated to say, easy to implement. What the OP describes ("use redis") is easy to say, complicated to implement. Not just in terms of human work time (setting up the redis machine and instance, connecting everything together), but also in terms of number of moving parts (more machines, more programs, etc.).
> We used tools that we're already using heavily in production and are comfortable with.
That's entirely fair, and what I figured was the most likely explanation.
Re: How We Built r/Place
#119Can someone link to the full resolution final image? Been trying to find it. Thanks!
Here you go: https://i.imgur.com/ajWiAYi.png . And for those interested, here's some additional stats: - The original announcement about /r/place: https://www.reddit.com/r/announcements/comments/62mesr/place... - Full timelapse of the canvas over the course all 72 hours : https://www.youtube.com/watch?v=XnRCZK3KjUY - Heatmap of all activity on the canvas over the full 72 hours : https://i.redd.it/20mghgkfwppy.png by…
awesome resources! Thanks!
Re: How We Built r/Place
#120What a fantastic writeup. I had some vague ideas regarding the challenges involved to build an application of such scale, but the article really makes it clear for everyone the amount of decision points encountered as well as why certain solutions were selected. I also like the way the article is broken down into the backend, API, frontend and mobile. This isolated approach really highlights the different struggles e…
That writeup may have taken more time than building it out. It seems quite thorough.
Edit: why am I getting down voted for praising a write-up?