For anyone just looking for the pub/sub functionality, I have been developing something that provides the functionality for PostgreSQL: https://github.com/supabase/realtime It's an Elixir server (Phoenix) that allows you to listen to changes in your database via websockets. Basically the Phoenix server listens to PostgreSQL's replication functionality, converts the byte stream into JSON, and then broadcasts over webs…
Do you know of anything like this for SQLite? I have a need for exactly this. Are there similar triggers?
Message DB: Event Store and Message Store for PostgreSQL
51–60 of 104 posts
Re: Message DB: Event Store and Message Store for PostgreSQL
#52On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?
At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…
I've been building systems on some pretty similar ideas for the last 5+ years, and have a similarly optimistic outlook on it. Just to add another datapoint here, we've been able to scale them up to 20-40x higher than your figure (a couple thousand jobs/sec) on a single (albeit beefy) MySQL instance. It definitely takes a fair bit of care & we certainly have some time invested in it, but it can be done.
I guess my services haven't had (gotten to? :-) ) to share databases with other services for quite some time -- they did when I started my job! -- but even with that concession, the benefits of DB guarantees have been amazing. There's nothing like a transaction for preserving your sanity.
The best tactic I've learned from these systems is to think really hard about the batching & partitioning behavior to keep msgs/sec high enough & transactions/sec low enough to stay afloat.
Re: Message DB: Event Store and Message Store for PostgreSQL
#53Very important bit to understand, that many smart people (including perhaps some in this HN discussion) don't know yet: the semantics of an event store are meaningfully different from the semantics of a pub/sub system. If you pick up a pub/sub system and start trying to build an event store based system architecture with a set of CQRS-ES-style followers, you will find you are reinventing a bunch of things quite painfully.
Here's what we did, a few years ago. In some ways it is similar to EventTide.
https://blog.oasisdigital.com/category/cqrs/
https://github.com/OasisDigital/nges
We picked up JGroups, a truly spectacular piece of open source technology, and used it to automatically wake up event stream listeners in a distributed way. (At the time there was some limitation of the NOTIFY mechanism in PG that sent us this direction.). With very little code around automated wake-up, we achieved low latency (and therefore fast automated testing) of an arbitrary group of event stream followers.
Re: Message DB: Event Store and Message Store for PostgreSQL
#54For anyone just looking for the pub/sub functionality, I have been developing something that provides the functionality for PostgreSQL: https://github.com/supabase/realtime It's an Elixir server (Phoenix) that allows you to listen to changes in your database via websockets. Basically the Phoenix server listens to PostgreSQL's replication functionality, converts the byte stream into JSON, and then broadcasts over webs…
[1]: https://github.com/hasura/graphql-engine/blob/master/event-t...
Re: Message DB: Event Store and Message Store for PostgreSQL
#55Earlier quoted context omitted.
At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…
For some perhaps-off-topic contrarianism, what do you make of Uncle Bob railing against database-oriented architectures? [0] (He takes a good few minutes to get his point across.) [0] https://youtu.be/o_TH-Y78tt4?t=2566
I've heard of him before but never really read anything much by him. Is this guy for real? He worked at one startup, got angry about a database, became a consultant, and screams about everyone being wrong about everything?
Re: Message DB: Event Store and Message Store for PostgreSQL
#56Earlier quoted context omitted.
Running a PG instance on DO for a startup is a really bad idea, I don't understand how can choose that over a managed solution. If you're a startup just use SNS / Kinesis / Google pub sub ect ...
Can you elaborate on why it is a really bad idea?
- Dealing with sec upgrade for PG and Linux
- Dealing with backups
- Dealing with HA, so settings up slaves / replica, then what do you do when something goes wrong? Do you manually SSH and do some magic?
- Network security / usernames / passwords
- ect ...
Clearly what startup don't want to do and usually lacks expertise into.
Re: Message DB: Event Store and Message Store for PostgreSQL
#57I have heard it said many times that it is a bad idea to implement message queues over a relational SQL database. e.g. "Databases suck for Messaging" https://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Nigh... http://mikehadlow.blogspot.com/2012/04/database-as-queue-ant... https://www.cloudamqp.com/blog/2015-11-23-why-is-a-database-... https://softwareengineering.stackexchange.com/questions/3514... I wonder if th…
In addition to what the sibling comments to this one said, Message DB solves a different problem than what queues solve. Message DB is a good fit in a microservice-based architecture, and the "micro" in "microservice" comes from concept of Dumb Pipes / Smart Endpoints ( https://martinfowler.com/articles/microservices.html#SmartEn... ). Message DB is a "dumb pipe," whereas queues and brokers fit into the "smart pipe"…
I don't know about that, I have seen many microservices that process data that comes in from messages on message queues; frequently that is a better design than receiving the data over synchronous http PUT or POST.
By "message queues" I mean AWS SNS/SQS and Azure event hubs. Would a SQL Db be "a good fit in a microservice-based architecture" replacement there?
or are you suggesting that such a microservice's first action on receiving a message from a queue would be to store it a local, private Message DB? That might make sense, I've seen enough tables that store a JSON blob already.
Re: Message DB: Event Store and Message Store for PostgreSQL
#58I have heard it said many times that it is a bad idea to implement message queues over a relational SQL database. e.g. "Databases suck for Messaging" https://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Nigh... http://mikehadlow.blogspot.com/2012/04/database-as-queue-ant... https://www.cloudamqp.com/blog/2015-11-23-why-is-a-database-... https://softwareengineering.stackexchange.com/questions/3514... I wonder if th…
The response given by @ethangarofolo does a good job of addressing the main points.
I would say that the slide deck linked is a bit sneaky. Sooner or later, in anything built with a computer, there's going to be polling. Higher-level libraries abstract the polling so that it appears as push (rather than pull) to the application developer. But under the hood, there's polling.
It should also be pointed out that any message queue with persistent messaging is built on a database. Even RabbitMQ's persistent messaging is built on a database (Mnesia).
In the end, a message store or event store can be used to model message queues, but the opposite is rarely true. One of the creators of the Event Store database once said to me something like, "What's a queue but a degenerate form of a stream".
A message store provides for patterns that are above and beyond message queues, like event sourcing, for example. Like Ethan mentioned, it's an application of "dumb pipes". It's in the same vein as Event Store or Kafka, rather than RabbitMQ, ActiveMQ, etc.
The critical difference is durability of messages. If you have an application that doesn't require durability of messages, then a plain old message queue or message broker technology may be a better choice for the situation.
In the end, polling is totally fine and totally manageable. What matters is that it's not done naively; that batching is intelligent and polling is only optionally triggered in the right circumstances and tuned based on batch processing cycle time and message arrival time.
Polling doesn't mean that the database will be "hammered" unless it's implemented that way.
Re: Message DB: Event Store and Message Store for PostgreSQL
#59I have heard it said many times that it is a bad idea to implement message queues over a relational SQL database. e.g. "Databases suck for Messaging" https://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Nigh... http://mikehadlow.blogspot.com/2012/04/database-as-queue-ant... https://www.cloudamqp.com/blog/2015-11-23-why-is-a-database-... https://softwareengineering.stackexchange.com/questions/3514... I wonder if th…
Yes, we disagree :) The response given by @ethangarofolo does a good job of addressing the main points. I would say that the slide deck linked is a bit sneaky. Sooner or later, in anything built with a computer, there's going to be polling. Higher-level libraries abstract the polling so that it appears as push (rather than pull) to the application developer. But under the hood, there's polling. It should also be poin…
Is that true? I am reminded of this essay that goes into a deep dive of what happens under the hood with http and the underlying network I/O
https://blog.stephencleary.com/2013/11/there-is-no-thread.ht...
And at the lowest level, the network card interrupts the CPU because it has finished reading or writing data.
> Some time after the write request started, the device finishes writing. It notifies the CPU via an interrupt.
Is that polling? It seems more like a push.
Re: Message DB: Event Store and Message Store for PostgreSQL
#60I have heard it said many times that it is a bad idea to implement message queues over a relational SQL database. e.g. "Databases suck for Messaging" https://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Nigh... http://mikehadlow.blogspot.com/2012/04/database-as-queue-ant... https://www.cloudamqp.com/blog/2015-11-23-why-is-a-database-... https://softwareengineering.stackexchange.com/questions/3514... I wonder if th…
Yes, we disagree :) The response given by @ethangarofolo does a good job of addressing the main points. I would say that the slide deck linked is a bit sneaky. Sooner or later, in anything built with a computer, there's going to be polling. Higher-level libraries abstract the polling so that it appears as push (rather than pull) to the application developer. But under the hood, there's polling. It should also be poin…
Good point that any system that is "persistent" has a data store by definition. The real question is that if it's a good fit to a SQL relational database.