Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Built r/Place

redditblog.com

Post reply on HN
Print view View original

211–220 of 255 posts

Re: How We Built r/Place

#211

Post by employee8000 »

josephg wrote: post #194
firebones wrote: post #190

Earlier quoted context omitted.

I wish I had enough spare cash to take you up on a bet that you could implement this in less than a weekend. Maybe HN can crowdfund paying you for two days at $1000/day to replicate, versus your time and a $3000 rebate if you can't.

My loved ones are out of town, so you're on! I'll do it for the fun of it and the experience. And because I still haven't built anything on top of kafka, but there's no time like now to fix that. The challenge is: Meet the main requirements of r/place: 1000x1000 image, a web based editor, 333+ edits / second, with an architecture that can scale to 100k simultaneous users (although that part will be hard to actually t…

Unless you stick with a single Kafka partition (which might not withstand the load) I don't think you can consume the events in exact chronological order.

Re: How We Built r/Place

#212

Post by easytiger »

wishinghand wrote: post #209
ChicagoBoy11 wrote: post #124

I love write-ups like this because they are such a nice contrast to the too-common comments on Reddit and HN where people claim that they could rebuild FB or Uber as a side project. Something as superficially trivial as the r/Place requires a tremendous amount of effort to run smoothly and there are countless gotchas and issues that you'd never even begin to consider unless you really tried to implement it yourself.…

> the too-common comments on Reddit and HN where people claim that they could rebuild FB or Uber as a side project. I'm pretty active on both. What threads/subreddits are you reading?

you should try r/unitedkingdom where all posters are pretty sure they could flawlessly redesign modern civilization

Re: How We Built r/Place

#213

Post by didibus »

josephg wrote: post #194
firebones wrote: post #190

Earlier quoted context omitted.

I wish I had enough spare cash to take you up on a bet that you could implement this in less than a weekend. Maybe HN can crowdfund paying you for two days at $1000/day to replicate, versus your time and a $3000 rebate if you can't.

My loved ones are out of town, so you're on! I'll do it for the fun of it and the experience. And because I still haven't built anything on top of kafka, but there's no time like now to fix that. The challenge is: Meet the main requirements of r/place: 1000x1000 image, a web based editor, 333+ edits / second, with an architecture that can scale to 100k simultaneous users (although that part will be hard to actually t…

Yes!! Love me a good show. I'll be following your tweets.

Re: How We Built r/Place

#215

Post by nathancahill »

easytiger wrote: post #212
wishinghand wrote: post #209

Earlier quoted context omitted.

> the too-common comments on Reddit and HN where people claim that they could rebuild FB or Uber as a side project. I'm pretty active on both. What threads/subreddits are you reading?

you should try r/unitedkingdom where all posters are pretty sure they could flawlessly redesign modern civilization

http://www.theonion.com/article/there-are-no-good-options-sy...

Re: How We Built r/Place

#216

Post by nathan_f77 »

This is fantastic. I learned a lot, and it seems like they nailed everything.

I really enjoyed the part about TypedArray and ArrayBuffer. And this might be a common thing to do, but I've never thought about using a CDN with an expiry time of 1 second, just to buffer lots of requests while still being close to real-time. That's brilliant.

Re: How We Built r/Place

#217

Post by scrollaway »

josephg wrote: post #205
xapata wrote: post #204

Earlier quoted context omitted.

You can simulate the users for load testing. Have them random walk from an arbitrary start point, changing a pixel every 10-20 minutes.

Simulating hundreds to thousands of writes per second via post requests will be easy. Simulating 100k simultaneous readers of an eventstream will be much harder. I don't know of any tools which can load test that many readers. Do you have any ideas?

locust can spawn arbitrary scripts; you'd have to write your own eventstream reader, and simulating 100k of them will take a few machines concurrently load-testing, but i think this should be the easiest way to do it.

Re: How We Built r/Place

#218

Post by josephg »

employee8000 wrote: post #211
josephg wrote: post #194

Earlier quoted context omitted.

My loved ones are out of town, so you're on! I'll do it for the fun of it and the experience. And because I still haven't built anything on top of kafka, but there's no time like now to fix that. The challenge is: Meet the main requirements of r/place: 1000x1000 image, a web based editor, 333+ edits / second, with an architecture that can scale to 100k simultaneous users (although that part will be hard to actually t…

Unless you stick with a single Kafka partition (which might not withstand the load) I don't think you can consume the events in exact chronological order.

A single kafka partition should be able to handle the load - it only needs to write 333 ops/second, and I could coalesce several edits together over half second windows if I wanted to make that more efficient. And I'll only have as many kafka consumers as I have frontend servers. So we're talking about 333 ops/second and like, 5 consumers of those operations. Somehow I think a single kafka partition should be able to manage the load :)

But there's no reason I couldn't use multiple partitions if I had to. To make it fully consistent I'd just need a consistent way to resolve conflicting concurrent writes on different partitions. Either adding a server timestamp to the messages, or using the partition number to pick a winner would work fine for this. It would just make the server logic a little more complicated.

Re: How We Built r/Place

#219

Post by kuschku »

nebabyte wrote: post #162
kuschku wrote: post #85

Earlier quoted context omitted.

Easily doable. On a 10$/month server I frequently run queries doing text operations over 120 million rows for fulltext search of an IRC client backlog. In 64ms. Without caching. Using PHP. It's definitely doable, but you'll need to heavily fine-tune your queries. My first one was at over 2 hours for the same.

> It's definitely doable, but you'll need to heavily fine-tune your queries Misrepresentation; then it's not actually over 120 million rows. You're basically encoding which subset to actually search in the query, rather than building a proper overall schema that trivializes queries.

No.

I mean stuff like using aggregation functions to let the database build a bitset out of them, and reading from every row a value into that bitset.

I mean stuff like CLUSTER table ON (pixel_y_index, pixel, x_index) to change the order in which they are stored.

Do those two optimizations alone and you improve speed massively.

Re: How We Built r/Place

#220

Post by Cthulhu_ »

eatitraw wrote: post #55

> We used our websocket service to publish updates to all the clients. I used /r/place from a few different browsers with a few different accounts, and they all seemed to have slightly different view of the same pixels. Was I the only one who experienced this problem? When /r/place experiment was still going, I assumed that they grouped updates in some sort of batches, but now it seems like they intended all users to…

I had the same thing, my client running out of sync (it seemed to be way behind) the "truth"; had to do a refresh to get it back up.
Post reply on HN