One problem I have with using postgres listen/notify as a general purpose message queue is that it requires polling (At least that was the case when I last looked at it). Of course you can use a blocking wrapper around the polling code but it still causes unnecessary roundtrips.
Million User Webchat with Full Stack Flux, React, Redis and PostgreSQL
21–30 of 41 posts
Re: Million User Webchat with Full Stack Flux, React, Redis and PostgreSQL
#22One problem I have with using postgres listen/notify as a general purpose message queue is that it requires polling (At least that was the case when I last looked at it). Of course you can use a blocking wrapper around the polling code but it still causes unnecessary roundtrips.
Huh? The entire point of listen/notify is exactly that you don't have to poll.
Re: Million User Webchat with Full Stack Flux, React, Redis and PostgreSQL
#23Earlier quoted context omitted.
but has this design been implemented to effectively handle something close to a million users Obviously not. Some of his numbers are off by an order of magnitude. E.g. he claims "10 million messages/sec" for a single redis instance. In reality redis tops out at well under one million messages/sec; http://redis.io/topics/benchmarks The design is almost comically bad (single source of truth for a "scalable" chat app?!)…
I agree. "Number of messages a single Node broker can handle: ~10k per second without JSON stringification memoization, ~100k per second with JSON stringification memoization (Nexus Flux socket.io does that for you)." <- Can't be possible. The numbers should be lower with JSON stringification/parsing instead of being 10x.
"Memoization" is the key word there. Apparently the author is expecting a lot of identical messages.
Re: Million User Webchat with Full Stack Flux, React, Redis and PostgreSQL
#24One problem I have with using postgres listen/notify as a general purpose message queue is that it requires polling (At least that was the case when I last looked at it). Of course you can use a blocking wrapper around the polling code but it still causes unnecessary roundtrips.
Your database connection is just a socket, so you can add that file descriptor to the set of file descriptors you are waiting IO on, if you are using a classic select/poll based system. See an example in the pscyopg2 docs here: http://initd.org/psycopg/docs/advanced.html#asynchronous-not...
Once that FD is active, you call the poll() method and your notify payload becomes available to you.
Re: Million User Webchat with Full Stack Flux, React, Redis and PostgreSQL
#25One problem I have with using postgres listen/notify as a general purpose message queue is that it requires polling (At least that was the case when I last looked at it). Of course you can use a blocking wrapper around the polling code but it still causes unnecessary roundtrips.
Huh? The entire point of listen/notify is exactly that you don't have to poll.
And yet in previous versions of libpq polling was the only way to use listen/notify. I'm not sure if it was a limitation of libpq or the server implementation itself: http://www.postgresql.org/docs/9.4/static/libpq-notify.html
It's still not a particularly simple interface as you have to check for notifications after every single SQL command if I understand it correctly.
Re: Million User Webchat with Full Stack Flux, React, Redis and PostgreSQL
#26One problem I have with using postgres listen/notify as a general purpose message queue is that it requires polling (At least that was the case when I last looked at it). Of course you can use a blocking wrapper around the polling code but it still causes unnecessary roundtrips.
Your database connection is just a socket, so you can add that file descriptor to the set of file descriptors you are waiting IO on, if you are using a classic select/poll based system. See an example in the pscyopg2 docs here: http://initd.org/psycopg/docs/advanced.html#asynchronous-not... Once that FD is active, you call the poll() method and your notify payload becomes available to you.
You are right. It seems that this was an issue with older versions of libpq: "In prior releases of libpq, the only way to ensure timely receipt of NOTIFY messages was to constantly submit commands"
http://www.postgresql.org/docs/9.4/static/libpq-notify.html
Re: Million User Webchat with Full Stack Flux, React, Redis and PostgreSQL
#27I suppose there is some value to this as a thought experiment but pretty much every tier in that architecture has breaking flaws. The most relevant being that anything that has an audience of 1 million users cannot run on an architecture that has single points of failure.
Re: Million User Webchat with Full Stack Flux, React, Redis and PostgreSQL
#28GameRanger has six million users, tens of thousands to hundreds of thousands of whom will be active simultaneously. His chat problem involved has moderate fan-out. Scott Kevill does it on a single machine (last time I checked) with hand-rolled C++ and close attention to the details of how the Linux networking stack works.
Very interesting, do you have links available?
Re: Million User Webchat with Full Stack Flux, React, Redis and PostgreSQL
#29GameRanger has six million users, tens of thousands to hundreds of thousands of whom will be active simultaneously. His chat problem involved has moderate fan-out. Scott Kevill does it on a single machine (last time I checked) with hand-rolled C++ and close attention to the details of how the Linux networking stack works.
I am also interested to know, Do you've any links on this?
Re: Million User Webchat with Full Stack Flux, React, Redis and PostgreSQL
#30The Flux terminology is confusing to me. It looks like the Observer pattern to me.
* Stores contain Observables
* Components (or Views) contain Observers
* Actions are Proxies
So the article is basically saying the Observer pattern is scalable, but uses the buzz-phrase "Full Stack Flux" instead. To make it even worse it is only a theoretical application of this pattern.