Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Message DB: Event Store and Message Store for PostgreSQL

blog.eventide-project.org

Post reply on HN
Print view View original

41–50 of 104 posts

Re: Message DB: Event Store and Message Store for PostgreSQL

#41

Post by ethangarofolo »

SahAssar wrote: post #15
ethangarofolo wrote: post #11

Earlier quoted context omitted.

The technologies you listed are message brokers , while Message DB is a message store . The former transport messages, and the latter is a database specialized for storing message data and sourcing system state from those messages. Kafka can, for example, move a lot of events around, but it isn't suitable for event sourcing for 2 key reasons. The first is that one generally has a separate stream for each entity in an…

Is this using NOTIFY/LISTEN to stream messages or some other way to get new messages as they arrive?

Good question. Consumers poll for updates. One of the stored functions in Message DB is designed for this very query.

Polling sounds very crude, but for the systems Message DB is designed for, it's a virtue. No back pressure problems, for example.

Re: Message DB: Event Store and Message Store for PostgreSQL

#42

Post by SideburnsOfDoom »

I have heard it said many times that it is a bad idea to implement message queues over a relational SQL database.

e.g.

"Databases suck for Messaging" https://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Nigh...

http://mikehadlow.blogspot.com/2012/04/database-as-queue-ant...

https://www.cloudamqp.com/blog/2015-11-23-why-is-a-database-...

https://softwareengineering.stackexchange.com/questions/3514...

I wonder if the designers of this system disagree, or if they did anything in particular to mitigate these issues?

Re: Message DB: Event Store and Message Store for PostgreSQL

#43

Post by tyre »

kiwicopple wrote: post #14

For anyone just looking for the pub/sub functionality, I have been developing something that provides the functionality for PostgreSQL: https://github.com/supabase/realtime It's an Elixir server (Phoenix) that allows you to listen to changes in your database via websockets. Basically the Phoenix server listens to PostgreSQL's replication functionality, converts the byte stream into JSON, and then broadcasts over webs…

Do you know of anything like this for SQLite? I have a need for exactly this. Are there similar triggers?

Re: Message DB: Event Store and Message Store for PostgreSQL

#44

Post by Tostino »

jtdev wrote: post #36
manigandham wrote: post #22

Earlier quoted context omitted.

NOTIFY just sends a pub/sub message. You still need a trigger or something else to actually call it on updates, and it has payload size limits.

Like...a trigger?

Yeah...triggers can have significant overhead though while sending a WAL stream to another server which will process it is super low overhead.

Re: Message DB: Event Store and Message Store for PostgreSQL

#45

Post by bradstewart »

SideburnsOfDoom wrote: post #42

I have heard it said many times that it is a bad idea to implement message queues over a relational SQL database. e.g. "Databases suck for Messaging" https://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Nigh... http://mikehadlow.blogspot.com/2012/04/database-as-queue-ant... https://www.cloudamqp.com/blog/2015-11-23-why-is-a-database-... https://softwareengineering.stackexchange.com/questions/3514... I wonder if th…

It depends heavily on what exactly you need out of a message queue. E.g. if extremely (extremely) high throughput is required and you don't mind dropped or lost messages, SQL may be a bad choice. But I've had a lot of success using Postgres-backed messaging.

Also your links are (mostly) from message queue companies who have a vested interest in your not using Postgres, or are from a time before LISTEN/NOTIFY and JSON(B) columns.

Re: Message DB: Event Store and Message Store for PostgreSQL

#46

Post by Thaxll »

sgk284 wrote: post #19
PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

Running a PG instance on DO for a startup is a really bad idea, I don't understand how can choose that over a managed solution.

If you're a startup just use SNS / Kinesis / Google pub sub ect ...

Re: Message DB: Event Store and Message Store for PostgreSQL

#47

Post by y4mi »

barrkel wrote: post #37
y4mi wrote: post #33

Earlier quoted context omitted.

That's incorrect. It's trivial If you have a clearly defined API with which to extract or insert data. It only starts to leak if everyone accesses the the DB through raw sql, mishmashing modules and creating insane dependencies. Give each module ownership to a table and demand to only access it through there and the backend storage system becomes irrelevant

If you put everything in the database, and use it for all it's able to do, as the comment I replied to suggested, it is practically inevitable under startup feature pressure conditions that total modularity won't hold. And in fact I think it would be irresponsible to pursue total modularity; it would court failure. I believe you're being deeply naive about modules owning single tables. You forego relational integrity…

You obviously need to write queries which span tables (I.e. joins).

I can see how you misunderstood me there however, it was admittedly poorly worded.

What I was talking about was basic data hygine as it's often called. If you need specific data, you define a clear way to get this data and only access it through that API. This can be a class, method or anything your language of choice prefers.

If you skip that step and directly access the DB everywhere in your code, you'll create another unmaintainable dumbsterfire as soon as your team goes beyond the initial programmers.

Re: Message DB: Event Store and Message Store for PostgreSQL

#48

Post by mumblemumble »

SideburnsOfDoom wrote: post #42

I have heard it said many times that it is a bad idea to implement message queues over a relational SQL database. e.g. "Databases suck for Messaging" https://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Nigh... http://mikehadlow.blogspot.com/2012/04/database-as-queue-ant... https://www.cloudamqp.com/blog/2015-11-23-why-is-a-database-... https://softwareengineering.stackexchange.com/questions/3514... I wonder if th…

There is definitely a performance ceiling if you use the RDMBS as your event store. I used to worry about that. Now that I've got a few years working at a very successful high frequency trading firm that did message queues in an RDBMS behind me, I don't worry about it so much anymore.

They didn't use it for everything; there was also a blindingly efficient homegrown messaging system for the stuff that needed to be really fast. But that was for fairly well-defined use cases. (Latency and concurrency were the key limiting factors.) By default, everything went into the RDBMS-based system. It was inherently more robust, it replaced a whole quagmire of design questions you have to make when using other messaging systems with ACID guarantees, and it made it a lot quicker & easier to do post-mortem analysis of any surprising things that happened in production.

Re: Message DB: Event Store and Message Store for PostgreSQL

#49

Post by tyri_kai_psomi »

Thaxll wrote: post #46
sgk284 wrote: post #19

Earlier quoted context omitted.

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

Running a PG instance on DO for a startup is a really bad idea, I don't understand how can choose that over a managed solution. If you're a startup just use SNS / Kinesis / Google pub sub ect ...

Can you elaborate on why it is a really bad idea?

Re: Message DB: Event Store and Message Store for PostgreSQL

#50

Post by ethangarofolo »

SideburnsOfDoom wrote: post #42

I have heard it said many times that it is a bad idea to implement message queues over a relational SQL database. e.g. "Databases suck for Messaging" https://www.rabbitmq.com/resources/RabbitMQ_Oxford_Geek_Nigh... http://mikehadlow.blogspot.com/2012/04/database-as-queue-ant... https://www.cloudamqp.com/blog/2015-11-23-why-is-a-database-... https://softwareengineering.stackexchange.com/questions/3514... I wonder if th…

In addition to what the sibling comments to this one said, Message DB solves a different problem than what queues solve. Message DB is a good fit in a microservice-based architecture, and the "micro" in "microservice" comes from concept of Dumb Pipes / Smart Endpoints (https://martinfowler.com/articles/microservices.html#SmartEn...). Message DB is a "dumb pipe," whereas queues and brokers fit into the "smart pipe" side of the divide.

Message DB is a message store, a database optimized for storing message data. The database doesn't track the status of anything---that responsibility falls to consumers of the data.

Message queues are fine pieces of technology when what you need is a message queue. Message DB, on the other hand, is used for event-sourced systems. "Event-sourced" in turn differs from merely "event-based." Since I've started building event-sourced systems, I haven't run across the need for message queues, but ymmv, and of course, like all humans, I too have my own hammer/nail biases.

Post reply on HN