Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Message DB: Event Store and Message Store for PostgreSQL

blog.eventide-project.org

Post reply on HN
Print view View original

31–40 of 104 posts

Re: Message DB: Event Store and Message Store for PostgreSQL

#31

Post by h91wka »

ethangarofolo wrote: post #11
PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

The technologies you listed are message brokers , while Message DB is a message store . The former transport messages, and the latter is a database specialized for storing message data and sourcing system state from those messages. Kafka can, for example, move a lot of events around, but it isn't suitable for event sourcing for 2 key reasons. The first is that one generally has a separate stream for each entity in an…

Incorrect. Kafka is a message _storage_. The server can be seen as a distributed persistent append-only log. All broker logic is encoded in the client. Source: maintainer of one of client implementations.

Re: Message DB: Event Store and Message Store for PostgreSQL

#32

Post by Dowwie »

PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

Antirez finally made Disque a Redis Module, too..

Re: Message DB: Event Store and Message Store for PostgreSQL

#33

Post by y4mi »

barrkel wrote: post #28
sgk284 wrote: post #19

Earlier quoted context omitted.

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

Of course, at the point where you do need to break out beyond a single DB, life is more painful because you've got dependencies on the database all over the place...

That's incorrect. It's trivial If you have a clearly defined API with which to extract or insert data.

It only starts to leak if everyone accesses the the DB through raw sql, mishmashing modules and creating insane dependencies.

Give each module ownership to a table and demand to only access it through there and the backend storage system becomes irrelevant

Re: Message DB: Event Store and Message Store for PostgreSQL

#34

Post by kiwicopple »

gigatexal wrote: post #21
sgk284 wrote: post #19

Earlier quoted context omitted.

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

Fascinating, was there ever more written about your tech stack?

Not OP, but we do something similar it our startup. PostgREST (http://postgrest.org) definitely helps if you want to go down this path as it exposes everything in the database (including functions etc) via a RestFUL api.

A bit outdated now, but some details here: https://paul.copplest.one/blog/nimbus-tech-2019-04.html#tech...

Re: Message DB: Event Store and Message Store for PostgreSQL

#35

Post by GordonS »

adverbly wrote: post #4

Looks cool. I can't find any performance or benchmarks though. Anyone care to guess/ballpark what kind of write speeds single nodes might be able to handle?

I got stuck looking for some perf figures too. I realise they are going to vary widely by hardware, batch size, message size etc, but it would be really useful to have a comparison against, for example, a single RabbitMQ instance.

Re: Message DB: Event Store and Message Store for PostgreSQL

#36

Post by jtdev »

manigandham wrote: post #22
ralusek wrote: post #17

Earlier quoted context omitted.

Is this better than postgres's native NOTIFY?

NOTIFY just sends a pub/sub message. You still need a trigger or something else to actually call it on updates, and it has payload size limits.

Like...a trigger?

Re: Message DB: Event Store and Message Store for PostgreSQL

#37

Post by barrkel »

y4mi wrote: post #33
barrkel wrote: post #28

Earlier quoted context omitted.

Of course, at the point where you do need to break out beyond a single DB, life is more painful because you've got dependencies on the database all over the place...

That's incorrect. It's trivial If you have a clearly defined API with which to extract or insert data. It only starts to leak if everyone accesses the the DB through raw sql, mishmashing modules and creating insane dependencies. Give each module ownership to a table and demand to only access it through there and the backend storage system becomes irrelevant

If you put everything in the database, and use it for all it's able to do, as the comment I replied to suggested, it is practically inevitable under startup feature pressure conditions that total modularity won't hold. And in fact I think it would be irresponsible to pursue total modularity; it would court failure.

I believe you're being deeply naive about modules owning single tables. You forego relational integrity and almost the whole power of relational algebra if you take that approach. That's heaps of functionality to leave behind when you and one or two other guys need to crank out a feature a day.

Re: Message DB: Event Store and Message Store for PostgreSQL

#38

Post by BenoitEssiambre »

sgk284 wrote: post #19
PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

To add to this, the numerous, impossible to solve issues that happen when your data starts hopping all over the network are proven mathematically:

https://groups.csail.mit.edu/tds/papers/Lynch/podc89.pdf

Re: Message DB: Event Store and Message Store for PostgreSQL

#39

Post by MaxBarraclough »

sgk284 wrote: post #19
PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

For some perhaps-off-topic contrarianism, what do you make of Uncle Bob railing against database-oriented architectures? [0] (He takes a good few minutes to get his point across.)

[0] https://youtu.be/o_TH-Y78tt4?t=2566

Re: Message DB: Event Store and Message Store for PostgreSQL

#40

Post by ethangarofolo »

barrkel wrote: post #28
sgk284 wrote: post #19

Earlier quoted context omitted.

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

Of course, at the point where you do need to break out beyond a single DB, life is more painful because you've got dependencies on the database all over the place...

Message DB happens to be implemented using a RDBMS, but the streams in it end up being very clear partition points. Some thought would be required to move data to a different database, but an event-sourced model isn't the same as coupling through a traditional RDBMS schema.

Edit: Fixed a typo

Post reply on HN