How We Built r/Place

Post by mozumder » Sat, Apr 15, 2017, 9:00 AM UTC

developer2 wrote: post #239

mozumder wrote: post #30

Earlier quoted context omitted.

I'd most likely have two tables - one for user activity and one for each pixel (1 million rows only in that table). Selecting a million rows from that pixel table might be 200ms or whatever. I'd still have Redis cache, though, since you're getting 100ms.

Consider exactly what you are proposing. One table to store the entire history (one billion or more rows). A second denormalized table, whether updated at the application layer or via triggers, to store the most recent update to each of the one million cells (1000x1000 pixel grid = one million data points). The simple fact of introducing a one-million-row read for the latest data of each "pixel cell" is fairly insane…

  place=> explain analyze select * from board_bitmap ;
                                                        QUERY PLAN                                                       
  -----------------------------------------------------------------------------------------------------------------------
   Seq Scan on board_bitmap  (cost=0.00..14425.00 rows=1000000 width=6) (actual time=0.009..57.295 rows=1000000 loops=1)
   Planning time: 0.160 ms
   Execution time: 90.510 ms
  (3 rows)

Post by svarrall » Sat, Apr 15, 2017, 10:06 AM UTC

Post by manigandham » Sat, Apr 15, 2017, 10:48 PM UTC

atombender wrote: post #138

d23 wrote: post #84

Earlier quoted context omitted.

Yeah, we went into it a bit in the "What We Learned" section, but that was most likely during the time we were having issues with RabbitMQ. I believe it was mostly fixed later on, but either way, we found a new pain point in our system we can now work on.

Surprised you're using RabbitMQ. It's one of those things which work great until they don't (clustering is particularly bad), and then you have almost zero insight into the issue, and have to resort to the Pivotal mailing list. Have you looked at NATS at all? We're using it as a message bus for one app and it's been fantastic. It is, however, an in-memory queue, and the current version cannot replace Rabbit for queue…

Post by manigandham » Sat, Apr 15, 2017, 11:02 PM UTC

d23 wrote: post #84

eatitraw wrote: post #55

> We used our websocket service to publish updates to all the clients. I used /r/place from a few different browsers with a few different accounts, and they all seemed to have slightly different view of the same pixels. Was I the only one who experienced this problem? When /r/place experiment was still going, I assumed that they grouped updates in some sort of batches, but now it seems like they intended all users to…

Yeah, we went into it a bit in the "What We Learned" section, but that was most likely during the time we were having issues with RabbitMQ. I believe it was mostly fixed later on, but either way, we found a new pain point in our system we can now work on.

Post by ralfd » Sun, Apr 16, 2017, 12:07 PM UTC

gramstrong wrote: post #77

writeslowly wrote: post #9

I thought it was interesting that one of their requirements was to provide an API that was easy to use for both bots and visualization tools. I remember reading some speculation when this was running that r/place was intentionally easy to interface with bots, while there were also complaints that the whole thing had been taken over by bots near the end.

Without bots, I doubt that /r/place would have been very interesting. It's a nice thought that a million random strangers can be cohesive without automation, but for some reason I don't find that to be particularly realistic..