Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Built r/Place

redditblog.com

Post reply on HN
Print view View original

161–170 of 255 posts

Re: How We Built r/Place

#161

Post by fernandotakai »

lobster_johnson wrote: post #149
fernandotakai wrote: post #142

Earlier quoted context omitted.

i've been using rabbitmq heavily (as in, the whole infrastructure is based on two rabbitmq servers) for a long time and i've never seen it fail. tbh, i never used clustering (because it's one of the shittiest clustering implementations i've ever seen) but we do use two servers (publishers connect to one randomly and consumers connect to both) and it seems to handle millions of messages without any issues. of all serv…

RabbitMQ is decent if you don't use clustering (which, I agree, is shitty). I have some quibbles with the non-clustered parts, but nothing big. Right now, the main annoyance is that it's impossible, as far as I understand, to limit its memory usage. You can set a "VM high watermark" and some other things, but beyond that, it will — much like, say, Elasticsearch — use a large amount of mysterious memory that you have…

yeah we have dedicated machines to rabbitmq because it's basically memory hungry. but i like it that way because it's only going to crash if the machine crashes.

Re: How We Built r/Place

#162

Post by nebabyte »

kuschku wrote: post #85
hashhar wrote: post #25

Earlier quoted context omitted.

And what about adding a million rows to the RDBMS and querying it in under 100ms (which Redis allowed them to do). I will say that it was an implementation bug which doesn't warrant the swapping out of entire data storage layers.

Easily doable. On a 10$/month server I frequently run queries doing text operations over 120 million rows for fulltext search of an IRC client backlog. In 64ms. Without caching. Using PHP. It's definitely doable, but you'll need to heavily fine-tune your queries. My first one was at over 2 hours for the same.

> It's definitely doable, but you'll need to heavily fine-tune your queries

Misrepresentation; then it's not actually over 120 million rows. You're basically encoding which subset to actually search in the query, rather than building a proper overall schema that trivializes queries.

Re: How We Built r/Place

#163

Post by [deleted] »

biot wrote: post #146

> At the peak of r/place the websocket service was > transmitting over 4 gbps (150 Mbps per instance > and 24 instances). What does Reddit use for serving up this much websocket traffic? Something open source, or is it custom built?

[deleted]

Re: How We Built r/Place

#164

Post by 77pt77 »

justintbassett wrote: post #69
aw3c2 wrote: post #68

Earlier quoted context omitted.

> Oops! We can't find that page.

my fault :). I have to get a few things ready for a public release of more data

Is this data:

https://www.reddit.com/r/place/comments/6396u5/rplace_archiv...

Complete?

I've been working with that.

Re: How We Built r/Place

#165

Post by corford »

mozumder wrote: post #2

> We actually had a race condition here that allowed users to place multiple tiles at once. There was no locking around the steps 1-3 so simultaneous tile draw attempts could all pass the check at step 1 and then draw multiple tiles at step 2. This is why you use a proper database. I'd probably add a Postgres table to record all user activity, and use that to lock out users for 5 minutes as an initial filter. Have tr…

Or just use redis for everything :) One instance for the bitfield, one for atomic locks (done with a lua script) and one for tile data (with a few slaves for reads). Simple and independently scaleable.

Re: How We Built r/Place

#166

Post by d23 »

atombender wrote: post #138
d23 wrote: post #84

Earlier quoted context omitted.

Yeah, we went into it a bit in the "What We Learned" section, but that was most likely during the time we were having issues with RabbitMQ. I believe it was mostly fixed later on, but either way, we found a new pain point in our system we can now work on.

Surprised you're using RabbitMQ. It's one of those things which work great until they don't (clustering is particularly bad), and then you have almost zero insight into the issue, and have to resort to the Pivotal mailing list. Have you looked at NATS at all? We're using it as a message bus for one app and it's been fantastic. It is, however, an in-memory queue, and the current version cannot replace Rabbit for queue…

Hmm, I'll take a look at that. For websockets, non-durability seems like a fine tradeoff, so it sounds interesting. Thanks!

Re: How We Built r/Place

#167

Post by Matheus28 »

nitwit005 wrote: post #39

Given the scale described, it sounds like they could have had a single machine that held the data in memory and periodically flushed to disk/DB to support failing over to a standby.

I would be slightly more careful and just use a cluster of servers with a simple consensus algorithm (like raft). A simple C++ server with a raft library plus uWebSockets should be able to handle a lot of load.

Re: How We Built r/Place

#169

Post by lobster_johnson »

d23 wrote: post #166
atombender wrote: post #138

Earlier quoted context omitted.

Surprised you're using RabbitMQ. It's one of those things which work great until they don't (clustering is particularly bad), and then you have almost zero insight into the issue, and have to resort to the Pivotal mailing list. Have you looked at NATS at all? We're using it as a message bus for one app and it's been fantastic. It is, however, an in-memory queue, and the current version cannot replace Rabbit for queue…

Hmm, I'll take a look at that. For websockets, non-durability seems like a fine tradeoff, so it sounds interesting. Thanks!

Note that NATS is currently pub/sub, which is a "if a tree falls in the forest" situation. Messages don't go anywhere if nobody is subscribing.

So it's awesome for realtime firehose-type use cases where a websocket client connects, receives messages (every client gets all the messages, although NATS also supports load-balanced fanout) for a while, then eventually disconnects.

NATS is ridiculously fast [1], too.

There's an add-on currently in beta, NATS Streaming [1], which [2] has durability, acking/redelivery and replay, so covers most of what you get from both RabbitMQ and Kafka. It looks very promising.

[1] http://bravenewgeek.com/tag/nats/

[2] https://nats.io/documentation/streaming/nats-streaming-intro...

Re: How We Built r/Place

#170

Post by andrewvc »

gooeyblob wrote: post #150
andrewvc wrote: post #113

Earlier quoted context omitted.

I agree that machines go down, but there are sane (and safe!) ways to build this sort of thing without adding in cassandra and Redis. Additionally, the max placement rate of 333/s is reaaaaally slow! Maybe that's due to the websocket frontends, not the DB, but, that doesn't mean that's the most obvious way to build it. The crux of the problem is that they need to mutate a relatively tiny amount of memory and have a r…

> The database would be a custom program that would: Creating a "custom database program" is not a small task. We like to use boring technologies that we know work well. We were already using Cassandra, had some experience with Redis, and had a lot of confidence in our CDN.

Well, in your article you mentioned that you tried to use Cassandra for one task and had to jettison it because of unexpected performance problems. You had to re-approach the problem with a whole other DB. I would say contradicts the point you're making.

I'm not arguing that most problems need a custom database, only a minority do. I'd say that this problem is borderline on which direction to go.

Databases are very leaky abstractions as you all discovered. The nice thing about custom code is that you don't have leaky abstractions. The bad thing about custom code is that you have a large new untested surface area.

In the case of your application the requirements are so minimal, a bitfield plus a log, I'd say its a wash.

Programmers today forget that things like flat files exist and are useful. It's a shame, because you wind up with situations where people just assume they need a giant distributed datastore for everything.

What you're doing in that case is trading architectural complexity for code complexity. Now, if its the case that all data in your org goes in one data store to keep things consistent, great, that makes sense. But for a one off app I just don't buy it.

Post reply on HN