Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Built r/Place

redditblog.com

Post reply on HN
Print view View original

141–150 of 255 posts

Re: How We Built r/Place

#141

Post by emorse »

subkamran wrote: post #109
brianwawok wrote: post #103

Earlier quoted context omitted.

What is the learning curve on a game engine for not game engine developers? They used tools they knew and knew how to scale. Almost always the tool you know is better than the perfect tool. They know redis and websockets, and they made it work. Beats using some engine know one on staff has ever touched?

I guess it depends on the engine, or maybe that was a rhetorical question. It doesn't take a month, or even a week, though--maybe to master them but not to learn the API. We've worked hard on our little project to design our engine for non-game developers or veterans, it's only a few lines of code to move a square, or draw an image, for example. But I understand the sentiment, especially for a one-time project.

I don't know, they had several well defined constraints and you don't know what a library is doing under the covers without spending some time with it. A proof of concept engine exploration is different than an app with 100,000 simultaneous users. I wonder how many business days they spent on this thing.

Re: How We Built r/Place

#142

Post by fernandotakai »

atombender wrote: post #138
d23 wrote: post #84

Earlier quoted context omitted.

Yeah, we went into it a bit in the "What We Learned" section, but that was most likely during the time we were having issues with RabbitMQ. I believe it was mostly fixed later on, but either way, we found a new pain point in our system we can now work on.

Surprised you're using RabbitMQ. It's one of those things which work great until they don't (clustering is particularly bad), and then you have almost zero insight into the issue, and have to resort to the Pivotal mailing list. Have you looked at NATS at all? We're using it as a message bus for one app and it's been fantastic. It is, however, an in-memory queue, and the current version cannot replace Rabbit for queue…

i've been using rabbitmq heavily (as in, the whole infrastructure is based on two rabbitmq servers) for a long time and i've never seen it fail.

tbh, i never used clustering (because it's one of the shittiest clustering implementations i've ever seen) but we do use two servers (publishers connect to one randomly and consumers connect to both) and it seems to handle millions of messages without any issues.

of all servers i've ever used, rabbitmq is by far the most stable (together with ejabberd).

Re: How We Built r/Place

#143

Post by fernandotakai »

rhizome wrote: post #34
andoon wrote: post #27

Earlier quoted context omitted.

Pages take a long time to generate all day, but during peak hours they take a minimum of 4 seconds each (depends on what page you're loading, if it's got lots of comments, etc), and many times they simply timeout. The engineers at reddit have been unable thus far to fix it.

This isn't my experience.

same. reddit was really bad some good years ago, but it's quite stable now.

Re: How We Built r/Place

#144

Post by josephg »

eblanshey wrote: post #115
seankimdesign wrote: post #75

What a fantastic writeup. I had some vague ideas regarding the challenges involved to build an application of such scale, but the article really makes it clear for everyone the amount of decision points encountered as well as why certain solutions were selected. I also like the way the article is broken down into the backend, API, frontend and mobile. This isolated approach really highlights the different struggles e…

Agreed, great write-up. Anyone have other recommended links to write-ups about specific challenges and how to make them scale?

http://highscalability.com/ frequently features content like this. The archives are a treasure trove of practical software architecture wisdom.

Re: How We Built r/Place

#145

Post by joenot443 »

DanHulton wrote: post #37
ag_47 wrote: post #28

Now I'm curious, Are there any websites that do something similar to /r/place? (hackathon idea?) Also, reminds be of the million dollar front page [1]. [1] https://en.wikipedia.org/wiki/The_Million_Dollar_Homepage

A long time ago, I built http://www.ipaidthemost.com/ , which is kinda related, at least to TMDH anyhow. Far, far less collaborative than /r/place, but similar in terms of staking out ownership.

I love it. Wish this had been my idea.

Re: How We Built r/Place

#146

Post by biot » 

  > At the peak of r/place the websocket service was
  > transmitting over 4 gbps (150 Mbps per instance
  > and 24 instances).
What does Reddit use for serving up this much websocket traffic? Something open source, or is it custom built?

Re: How We Built r/Place

#148

Post by Viper007Bond »

Ajedi32 wrote: post #70
calosa wrote: post #61

If any data science-y folks want to work with the raw data, you can find it here... https://data.world/justintbassett/place-events

"Oops! We can't find that page."

> my fault :). I have to get a few things ready for a public release of more data

https://news.ycombinator.com/item?id=14110094

Re: How We Built r/Place

#149

Post by lobster_johnson »

fernandotakai wrote: post #142
atombender wrote: post #138

Earlier quoted context omitted.

Surprised you're using RabbitMQ. It's one of those things which work great until they don't (clustering is particularly bad), and then you have almost zero insight into the issue, and have to resort to the Pivotal mailing list. Have you looked at NATS at all? We're using it as a message bus for one app and it's been fantastic. It is, however, an in-memory queue, and the current version cannot replace Rabbit for queue…

i've been using rabbitmq heavily (as in, the whole infrastructure is based on two rabbitmq servers) for a long time and i've never seen it fail. tbh, i never used clustering (because it's one of the shittiest clustering implementations i've ever seen) but we do use two servers (publishers connect to one randomly and consumers connect to both) and it seems to handle millions of messages without any issues. of all serv…

RabbitMQ is decent if you don't use clustering (which, I agree, is shitty). I have some quibbles with the non-clustered parts, but nothing big.

Right now, the main annoyance is that it's impossible, as far as I understand, to limit its memory usage. You can set a "VM high watermark" and some other things, but beyond that, it will — much like, say, Elasticsearch — use a large amount of mysterious memory that you have no control over. You can't just say "use 1GB and nothing more", which is problematic on Kubernetes where you want to pack things a bit tightly. This happens even if all the queues are marked as durable.

Re: How We Built r/Place

#150

Post by gooeyblob »

andrewvc wrote: post #113
eric_h wrote: post #92

Earlier quoted context omitted.

> Why use Redis and multiple machines instead of keeping it in RAM on a single machine? Because machines go down. If you don't expect your hardware to fail at the most inopportune time, you'll be screwed when (not if) it happens.

I agree that machines go down, but there are sane (and safe!) ways to build this sort of thing without adding in cassandra and Redis. Additionally, the max placement rate of 333/s is reaaaaally slow! Maybe that's due to the websocket frontends, not the DB, but, that doesn't mean that's the most obvious way to build it. The crux of the problem is that they need to mutate a relatively tiny amount of memory and have a r…

> The database would be a custom program that would:

Creating a "custom database program" is not a small task.

We like to use boring technologies that we know work well. We were already using Cassandra, had some experience with Redis, and had a lot of confidence in our CDN.

Post reply on HN