Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Message DB: Event Store and Message Store for PostgreSQL

blog.eventide-project.org

Post reply on HN
Print view View original

11–20 of 104 posts

Re: Message DB: Event Store and Message Store for PostgreSQL

#11

Post by ethangarofolo »

PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

The technologies you listed are message brokers, while Message DB is a message store. The former transport messages, and the latter is a database specialized for storing message data and sourcing system state from those messages. Kafka can, for example, move a lot of events around, but it isn't suitable for event sourcing for 2 key reasons.

The first is that one generally has a separate stream for each entity in an event-sourced system. Streams are sort of like topics in Kafka, but it would be quite challenging to, say, make a topic per user in Kafka. The second is Kafka's lack of optimistic concurrency support (see https://issues.apache.org/jira/browse/KAFKA-2260). The decision to not support expected offsets makes perfect sense for what Kafka is, but it does make it unsuitable for event sourcing.

Being built on top of Postgres, Message DB gives you access to event sourcing semantics using a familiar database technology.

We use Message DB in our production systems, and I'd be happy to talk more about it if you have other questions. We've found it very reliable.

As a disclaimer, I'm listed as a contributor at the Eventide Project, the project Message DB was extracted from, though I did not write any of the code behind Message DB.

Re: Message DB: Event Store and Message Store for PostgreSQL

#13

Post by iamspoilt »

ethangarofolo wrote: post #11
PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

The technologies you listed are message brokers , while Message DB is a message store . The former transport messages, and the latter is a database specialized for storing message data and sourcing system state from those messages. Kafka can, for example, move a lot of events around, but it isn't suitable for event sourcing for 2 key reasons. The first is that one generally has a separate stream for each entity in an…

I think the biggest difference is that it’s a Message Store as highlighted by the preceding comment.

Some clarity by example: In financial systems, it’s extremely important to keep track of all the transactions between your microservices (if your architecture is based on that). You could potentially lose a message delivered to you via a broker if your service fails to write it to a persistent storage. If the producer of that message never stored that message or it was produced on wire and transmitted, there is no way to recover it anymore. A system designed around a Message Store can mitigate such problems. You can build that similar architecture with brokers as well but for each of your application, you will have to implement something analogous to a Message Store to handle idempotency and things like that.

Re: Message DB: Event Store and Message Store for PostgreSQL

#14

Post by kiwicopple »

For anyone just looking for the pub/sub functionality, I have been developing something that provides the functionality for PostgreSQL: https://github.com/supabase/realtime

It's an Elixir server (Phoenix) that allows you to listen to changes in your database via websockets.

Basically the Phoenix server listens to PostgreSQL's replication functionality, converts the byte stream into JSON, and then broadcasts over websockets. The beauty of listening to the replication functionality is that you can make changes to your database from anywhere - your api, directly in the DB, via a console etc - and you will still receive the changes via websockets.

I'd have to dig into their implementation to verify, but the main benefit over OP is scalability - i believe their implementation is tightly coupled to postgres, but with this you could scale the Elixir servers without any additional load on your DB

It's still in very early stages, although I am using it in production at my company and will work on it full time starting Jan

Re: Message DB: Event Store and Message Store for PostgreSQL

#15

Post by SahAssar »

ethangarofolo wrote: post #11
PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

The technologies you listed are message brokers , while Message DB is a message store . The former transport messages, and the latter is a database specialized for storing message data and sourcing system state from those messages. Kafka can, for example, move a lot of events around, but it isn't suitable for event sourcing for 2 key reasons. The first is that one generally has a separate stream for each entity in an…

Is this using NOTIFY/LISTEN to stream messages or some other way to get new messages as they arrive?

Re: Message DB: Event Store and Message Store for PostgreSQL

#16

Post by SahAssar »

fiatjaf wrote: post #9

"for"? I feel like this should be "in PostgreSQL". I thought this was something that was used internally by Postgres.

"for" is pretty commonly used in this way. See for example https://koajs.com/ and https://rocket.rs/ which both say "web framework for X"

Re: Message DB: Event Store and Message Store for PostgreSQL

#17

Post by ralusek »

kiwicopple wrote: post #14

For anyone just looking for the pub/sub functionality, I have been developing something that provides the functionality for PostgreSQL: https://github.com/supabase/realtime It's an Elixir server (Phoenix) that allows you to listen to changes in your database via websockets. Basically the Phoenix server listens to PostgreSQL's replication functionality, converts the byte stream into JSON, and then broadcasts over webs…

Is this better than postgres's native NOTIFY?

Re: Message DB: Event Store and Message Store for PostgreSQL

#18

Post by kiwicopple »

ralusek wrote: post #17
kiwicopple wrote: post #14

For anyone just looking for the pub/sub functionality, I have been developing something that provides the functionality for PostgreSQL: https://github.com/supabase/realtime It's an Elixir server (Phoenix) that allows you to listen to changes in your database via websockets. Basically the Phoenix server listens to PostgreSQL's replication functionality, converts the byte stream into JSON, and then broadcasts over webs…

Is this better than postgres's native NOTIFY?

I actually started with NOTIFY but the payload has a limit of 8000 bytes. Postgres throws an error for anything longer.

Re: Message DB: Event Store and Message Store for PostgreSQL

#19

Post by sgk284 »

PankajGhosh wrote: post #6

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical.

It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services).

We had several services running, but any notion of persistent state was stored in a single database. We didn't have particularly high demands, but processed ~140M jobs per month in addition to our normal query load.

Postgres handled this like a champ, had far lower p50 and p95 latency than SNS/SQS, and rarely went above single-digit percentage CPU usage on a fairly cheap DigitalOcean box.

I've worked at a lot of big tech companies (Google, Microsoft, Twitter, Salesforce) in many varieties of giant distributed systems and the most valuable lesson I've learned over and over again is: Distributed systems are hard, avoid them until you can't.

That's the best advice I can give to any startup. Until you're regularly sustaining 1,000's of TPS or have grown to dozens of TB of data, it is generally a distraction to even think about anything other than a single relational database.

Post reply on HN