Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Built r/Place

redditblog.com

Post reply on HN
Print view View original

81–90 of 255 posts

Re: How We Built r/Place

#81

Post by kuschku »

gramstrong wrote: post #77
writeslowly wrote: post #9

I thought it was interesting that one of their requirements was to provide an API that was easy to use for both bots and visualization tools. I remember reading some speculation when this was running that r/place was intentionally easy to interface with bots, while there were also complaints that the whole thing had been taken over by bots near the end.

Without bots, I doubt that /r/place would have been very interesting. It's a nice thought that a million random strangers can be cohesive without automation, but for some reason I don't find that to be particularly realistic..

Oh it is very easy. You just need subreddits with lots of very loyal people who even do frequent meetups, and lots of those subreddits.

And soon you get exactly what /r/place was.

Re: How We Built r/Place

#82

Post by tyrust »

foota wrote: post #63
tyrust wrote: post #48

Earlier quoted context omitted.

They prepared for the worst (peak 100k users). The "hope" that those 100k would be spread out was based on the statistical likelihood that these 100k wouldn't line up too much over a 5 minute period. I didn't follow /r/place that much, but I haven't read any complaints about latency or failures so it looks like they did just fine.

They say elsewhere that they were prepared for 100k updates all at once. Which is the worst. Edit: you're an sre, you probably have more experience planning these things than I ever will. I just can't help but think about what would happen if some trolls realized they could synchronize their updates and bring down the service. (Although based on the infrastructure it doesn't seem possible to cause lasting damage.)

Yeah, I understand what you're saying. It is conceivable that a bunch of people would coordinate their updates. To prevent new-comers from spoiling the fun, they only allowed users with accounts created before /r/place was launched to send updates.

Re: How We Built r/Place

#84

Post by d23 »

eatitraw wrote: post #55

> We used our websocket service to publish updates to all the clients. I used /r/place from a few different browsers with a few different accounts, and they all seemed to have slightly different view of the same pixels. Was I the only one who experienced this problem? When /r/place experiment was still going, I assumed that they grouped updates in some sort of batches, but now it seems like they intended all users to…

Yeah, we went into it a bit in the "What We Learned" section, but that was most likely during the time we were having issues with RabbitMQ. I believe it was mostly fixed later on, but either way, we found a new pain point in our system we can now work on.

Re: How We Built r/Place

#85

Post by kuschku »

hashhar wrote: post #25
mozumder wrote: post #2

> We actually had a race condition here that allowed users to place multiple tiles at once. There was no locking around the steps 1-3 so simultaneous tile draw attempts could all pass the check at step 1 and then draw multiple tiles at step 2. This is why you use a proper database. I'd probably add a Postgres table to record all user activity, and use that to lock out users for 5 minutes as an initial filter. Have tr…

And what about adding a million rows to the RDBMS and querying it in under 100ms (which Redis allowed them to do). I will say that it was an implementation bug which doesn't warrant the swapping out of entire data storage layers.

Easily doable. On a 10$/month server I frequently run queries doing text operations over 120 million rows for fulltext search of an IRC client backlog.

In 64ms.

Without caching.

Using PHP.

It's definitely doable, but you'll need to heavily fine-tune your queries. My first one was at over 2 hours for the same.

Re: How We Built r/Place

#86

Post by tupshin »

Our initial approach was to store the full board in a single row in Cassandra and each request for the full board would read that entire row.

This is the epitome of an anti-pattern .I sincerely hope that this approach was floated by somebody who had never used Cassandra before.

Even if individual requests were reasonably fast, you are sticking all of your data in a single partition, creating the hottest of hot spots and failing to leverage the scale out nature of your database.

Re: How We Built r/Place

#87

Post by pimeys »

stevekemp wrote: post #73
maaaats wrote: post #19

Interesting how big the Norwegian and Swedish flags got, given our small populations.

I thought the exact same thing, about the Finnish flag - complete with Moomins.

And the almost a dictator of a president Urho Kekkonen. I kind of understand this, I was born in that country...

Re: How We Built r/Place

#88

Post by eric_h »

tummybug wrote: post #57
avip wrote: post #26

Earlier quoted context omitted.

I can shed that 333 [something/s], for any uncomplicated something, is so little that a single-core 10YO laptop running a single-process non-blocking webserver (a-la node) could probably handle it, and likely x10 it.

Sure in a perfect scenario with a well behaving load test client. In production you often encounter scenarios which place unexpected load on your servers.

Ah yes, the self-inflicted unintentional DDoS attack that doesn't appear when you do a semi-idealized load test. I may or may not have been responsible for one of those once or twice in my life.

Re: How We Built r/Place

#90

Post by spyspy »

tupshin wrote: post #86

Our initial approach was to store the full board in a single row in Cassandra and each request for the full board would read that entire row. This is the epitome of an anti-pattern .I sincerely hope that this approach was floated by somebody who had never used Cassandra before. Even if individual requests were reasonably fast, you are sticking all of your data in a single partition, creating the hottest of hot spots…

This entire project is just an elaborate hack day project. There's no reason to fault them for trying new and interesting hacks to get it off the ground. They realized it wasn't the right method and moved on. End of story.
Post reply on HN