Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Message DB: Event Store and Message Store for PostgreSQL

blog.eventide-project.org

Post reply on HN
Print view View original

21–30 of 104 posts

Re: Message DB: Event Store and Message Store for PostgreSQL

#21

Post by gigatexal »

sgk284 wrote: post #19
PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

Fascinating, was there ever more written about your tech stack?

Re: Message DB: Event Store and Message Store for PostgreSQL

#22

Post by manigandham »

ralusek wrote: post #17
kiwicopple wrote: post #14

For anyone just looking for the pub/sub functionality, I have been developing something that provides the functionality for PostgreSQL: https://github.com/supabase/realtime It's an Elixir server (Phoenix) that allows you to listen to changes in your database via websockets. Basically the Phoenix server listens to PostgreSQL's replication functionality, converts the byte stream into JSON, and then broadcasts over webs…

Is this better than postgres's native NOTIFY?

NOTIFY just sends a pub/sub message. You still need a trigger or something else to actually call it on updates, and it has payload size limits.

Re: Message DB: Event Store and Message Store for PostgreSQL

#23

Post by manigandham »

sgk284 wrote: post #19
PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

That's great to read, and a perfect example of solving real problems instead of wasting time on exotic infrastructure.

The vast majority of companies and their scale will never go above a single midsize database server. All of the transactional, backup, and querying functionalities you list make it much more productive than using fancy AWS/cloud services.

Re: Message DB: Event Store and Message Store for PostgreSQL

#24

Post by davidgl »

PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

Depends on your requirements, and the message throughput, but we have had great success using a database as a message broker. We get atomic commits on jobs, audit history, database backup is all we need for recovery, and a less complex stack. We did use RabbitMQ, but replaced it with using SQL Server, and the sql notifications to make it efficient. Couldn't be happier, but we only do thousands of messages a day, not sure I'd use the same approach with millions.

Re: Message DB: Event Store and Message Store for PostgreSQL

#25

Post by simplecto »

sgk284 wrote: post #19
PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

> Distributed systems are hard, avoid them until you can't.

thank you. May I borrow this?

Re: Message DB: Event Store and Message Store for PostgreSQL

#26

Post by hashhar »

kiwicopple wrote: post #14

For anyone just looking for the pub/sub functionality, I have been developing something that provides the functionality for PostgreSQL: https://github.com/supabase/realtime It's an Elixir server (Phoenix) that allows you to listen to changes in your database via websockets. Basically the Phoenix server listens to PostgreSQL's replication functionality, converts the byte stream into JSON, and then broadcasts over webs…

Look at embedded Debezium. https://debezium.io/docs/embedded/

Very similar to what you are doing but instead of Websockets you can decide where to send your data. The core-debezium sends it to Kafka.

Re: Message DB: Event Store and Message Store for PostgreSQL

#27

Post by djrobstep »

sgk284 wrote: post #19
PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

We did a very similar thing where I used to work. Didn't want to bother with a separate task queue and wanted a consistent SQL interface for everything.

We wrote a very simple task queue (<500 lines of python) using a postgres table and inbuilt pub/sub. Ran millions of jobs, scheduled and unscheduled, without a hitch.

Re: Message DB: Event Store and Message Store for PostgreSQL

#28

Post by barrkel »

sgk284 wrote: post #19
PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

Of course, at the point where you do need to break out beyond a single DB, life is more painful because you've got dependencies on the database all over the place...

Re: Message DB: Event Store and Message Store for PostgreSQL

#29

Post by [deleted] »

ethangarofolo wrote: post #11
PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

The technologies you listed are message brokers , while Message DB is a message store . The former transport messages, and the latter is a database specialized for storing message data and sourcing system state from those messages. Kafka can, for example, move a lot of events around, but it isn't suitable for event sourcing for 2 key reasons. The first is that one generally has a separate stream for each entity in an…

[deleted]

Re: Message DB: Event Store and Message Store for PostgreSQL

#30

Post by kiwicopple »

hashhar wrote: post #26
kiwicopple wrote: post #14

For anyone just looking for the pub/sub functionality, I have been developing something that provides the functionality for PostgreSQL: https://github.com/supabase/realtime It's an Elixir server (Phoenix) that allows you to listen to changes in your database via websockets. Basically the Phoenix server listens to PostgreSQL's replication functionality, converts the byte stream into JSON, and then broadcasts over webs…

Look at embedded Debezium. https://debezium.io/docs/embedded/ Very similar to what you are doing but instead of Websockets you can decide where to send your data. The core-debezium sends it to Kafka.

I took a lot of inspiration from Debezium - at the time they required the wal2json plugin (not sure if that's still the case?). I didn't have an option to install the plugin and I mostly wanted a websocket-friendly implementation to replace Firebase.

And if I'm 100% honest I just wanted to make something i thought was cool

Post reply on HN