Live data from Hacker News

The realtime magic behind Moot and how we deliver all your instant notifications

moot.it

11–20 of 47 posts

Re: The realtime magic behind Moot and how we deliver all your instant notifications

#11
post #3

I followed it the whole way through until I got to the diagram of the way Moot handles channels compared to static channels. What does the diagram illustrate? It seems to be that the Moot version doesn't have central input. But it would all need to be filtered through something central, otherwise it is completely possible to miss a location the payload was intended to be delivered to.

That central location is Redis pub/sub. Multiple publishers publish to Redis and many JSON-RPC instances listen and then process the notifications to distribute. The diagram is more meant to illustrate that the notifications don't just blanket deliver to all subscribers rather than be an exactly technically accurate.

Realistically, it sounds like the 2nd diagram should have the middle section of the first diagram. As in, a single message/circle, gets "split" (or more accurately dup'd) and somewhat simultaneously delivered to the start of every 'channel'. The way it is now makes the appearance that a message isn't sent down a channel where the logic gate will not eventually deliver it. But that doesn't seem to be the case.

Re: The realtime magic behind Moot and how we deliver all your instant notifications

#12
post #7

Earlier quoted context omitted.

Yes we do see this as becoming a bottleneck. We are already working on a solution for that. I mention in the "What's Next" section about a bit more advanced routing using ZeroMQ instead of Redis. The problem is a difficult one to say the least. We're nowhere near the bottleneck now, but we do want to resolve this long before it is close to becoming an issue.

Could you please elaborate on how using ZeroMQ would solve your problem? I have a surprisingly similar setup and I'm also worried that this is going to be a bottleneck.

ZeroMQ doesn't solve the problem. ZeroMQ is flexible enough that we can build a solution. We're working on distributed routing and registration using ZeroMQ and Redis as it's data store.

The RPC servers register the tags and channels we distribute based on that data. Any ZeroMQ server can process and deliver any event to the end RPC server.

We'll probably do a detailed writeup when this is done. We plan on using this for all the infrastructure wiring when it's done.

Re: The realtime magic behind Moot and how we deliver all your instant notifications

#13
post #10

Are you going to open-source your json-rpc library? If so, is there a specific timeframe you have in mind? I'd really love to use this technology in my own projects.

Yes we intent to open source all of this. No timeframe in mind. We need to get through our beta roadmap before packaging up the various pieces we plan on open sourcing.

Re: The realtime magic behind Moot and how we deliver all your instant notifications

#15
post #10

Are you going to open-source your json-rpc library? If so, is there a specific timeframe you have in mind? I'd really love to use this technology in my own projects.

Yes we intent to open source all of this. No timeframe in mind. We need to get through our beta roadmap before packaging up the various pieces we plan on open sourcing.

Thank you so much, that's amazing! :)

Re: The realtime magic behind Moot and how we deliver all your instant notifications

#16
Ehm.. tens of millions of realtime events but no "www." alias?:

404 Not Found

Code: NoSuchWebsiteConfiguration

Message: The specified bucket does not have a website configuration

BucketName: www.moot.it

RequestId: xxxx

HostId: xxxxx

I didn't know what "Moot" was so I tried www.moot.it

Re: The realtime magic behind Moot and how we deliver all your instant notifications

#17
post #11

Earlier quoted context omitted.

That central location is Redis pub/sub. Multiple publishers publish to Redis and many JSON-RPC instances listen and then process the notifications to distribute. The diagram is more meant to illustrate that the notifications don't just blanket deliver to all subscribers rather than be an exactly technically accurate.

Realistically, it sounds like the 2nd diagram should have the middle section of the first diagram. As in, a single message/circle, gets "split" (or more accurately dup'd) and somewhat simultaneously delivered to the start of every 'channel'. The way it is now makes the appearance that a message isn't sent down a channel where the logic gate will not eventually deliver it. But that doesn't seem to be the case.

Yeah, yeah, I know: I'm just a graphic designer without a deep knowledge of the technology... But what happens in the diagram is still pretty close to what you describe, to be fair: messages duped and delivered simultaneously. Absolute accuracy would have made for a pretty complex illustration. Point taken anyway, I'll try harder next time!

Btw, did you notice it has blue heads? I thought those were cool.

Re: The realtime magic behind Moot and how we deliver all your instant notifications

#18
post #16

Ehm.. tens of millions of realtime events but no "www." alias?: 404 Not Found Code: NoSuchWebsiteConfiguration Message: The specified bucket does not have a website configuration BucketName: www.moot.it RequestId: xxxx HostId: xxxxx I didn't know what "Moot" was so I tried www.moot.it

Hmm I think you must have hit it while we were doing a deploy. Looks like the www alias points right to S3 instead of our CDN. The CDN would have hidden the bucket reconfiguration. Oops.

Will be updating the alias to point to the CDN. Thanks for catching that.

Re: The realtime magic behind Moot and how we deliver all your instant notifications

#20
post #7

Earlier quoted context omitted.

Could you please elaborate on how using ZeroMQ would solve your problem? I have a surprisingly similar setup and I'm also worried that this is going to be a bottleneck.

ZeroMQ doesn't solve the problem. ZeroMQ is flexible enough that we can build a solution. We're working on distributed routing and registration using ZeroMQ and Redis as it's data store. The RPC servers register the tags and channels we distribute based on that data. Any ZeroMQ server can process and deliver any event to the end RPC server. We'll probably do a detailed writeup when this is done. We plan on using this…

Please do a writeup when you've done this - would love to read about this - very relevant to my work as well.
Post reply on HN