Message DB: Event Store and Message Store for PostgreSQL • Hacker News Board

Re: Message DB: Event Store and Message Store for PostgreSQL

#61

Post by mumblemumble » Tue, Dec 17, 2019, 5:20 PM UTC

SideburnsOfDoom wrote: post #57

ethangarofolo wrote: post #50

Earlier quoted context omitted.

In addition to what the sibling comments to this one said, Message DB solves a different problem than what queues solve. Message DB is a good fit in a microservice-based architecture, and the "micro" in "microservice" comes from concept of Dumb Pipes / Smart Endpoints ( https://martinfowler.com/articles/microservices.html#SmartEn... ). Message DB is a "dumb pipe," whereas queues and brokers fit into the "smart pipe"…

> Message DB is a good fit in a microservice-based architecture I don't know about that, I have seen many microservices that process data that comes in from messages on message queues; frequently that is a better design than receiving the data over synchronous http PUT or POST. By "message queues" I mean AWS SNS/SQS and Azure event hubs. Would a SQL Db be "a good fit in a microservice-based architecture" replacement…

Take a look at that 2nd link in the grandparent post. I remember it as one that was often cited when this debate first went around the Internet half a decade or so ago.

The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed.

One could argue that that is the real antipattern. It'll hurt you big time if you're storing messages in a database table. But it'll hurt you even if you don't. For example, by tracking that kind of information inside of the queue, you're losing the ability to add a second listener without either affecting what messages are being seen by (and therefore the behavior of) the existing listener(s), or modifying the queue itself. Which you don't want to have to do any more than you have to on a critical piece of shared infrastructure like that.

It might be fine if it's an oldschool monolith that's just processing some sort of work queue on multiple threads. But, if you're doing microservices, you're probably trying to keep things more flexible than that, and want to allow them to evolve more independently.

Re: Message DB: Event Store and Message Store for PostgreSQL

#62

Post by sbellware » Tue, Dec 17, 2019, 5:21 PM UTC

SideburnsOfDoom wrote: post #59

sbellware wrote: post #58

Earlier quoted context omitted.

Yes, we disagree :) The response given by @ethangarofolo does a good job of addressing the main points. I would say that the slide deck linked is a bit sneaky. Sooner or later, in anything built with a computer, there's going to be polling. Higher-level libraries abstract the polling so that it appears as push (rather than pull) to the application developer. But under the hood, there's polling. It should also be poin…

> Sooner or later, in anything built with a computer, there's going to be polling. ... But under the hood, there's polling. Is that true? I am reminded of this essay that goes into a deep dive of what happens under the hood with http and the underlying network I/O https://blog.stephencleary.com/2013/11/there-is-no-thread.ht... And at the lowest level, the network card interrupts the CPU because it has finished readin…

Anything that processes a signal checks if a signal has been received. It's no so black-and-white at the level of electricity, but higher-level things at the level of durable message queues check for new signals, even if those signals arrive via "push".

Re: Message DB: Event Store and Message Store for PostgreSQL

#63

Post by sbellware » Tue, Dec 17, 2019, 5:28 PM UTC

mumblemumble wrote: post #61

SideburnsOfDoom wrote: post #57

Earlier quoted context omitted.

> Message DB is a good fit in a microservice-based architecture I don't know about that, I have seen many microservices that process data that comes in from messages on message queues; frequently that is a better design than receiving the data over synchronous http PUT or POST. By "message queues" I mean AWS SNS/SQS and Azure event hubs. Would a SQL Db be "a good fit in a microservice-based architecture" replacement…

Take a look at that 2nd link in the grandparent post. I remember it as one that was often cited when this debate first went around the Internet half a decade or so ago. The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed. One could argue that that is the real antipattern. It'll hurt you big tim…

> The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed. One could argue that that is the real antipattern.

Indeed.

Lots of good writing on the tradeoffs, for example: https://sookocheff.com/post/messaging/dissecting-sqs-fifo-qu...

At the risk of beating the point to death, the transition from SOA to Microservices is marked by the transition of smart pipes to dumb pipes, and dumb pipes have no knowledge of whether a message has been processed. In the microservices era, the basic presumption is that the transport has no knowledge of the state of the message processors' progress through a queue.

Re: Message DB: Event Store and Message Store for PostgreSQL

#64

Post by SideburnsOfDoom » Tue, Dec 17, 2019, 5:44 PM UTC

mumblemumble wrote: post #61

SideburnsOfDoom wrote: post #57

Earlier quoted context omitted.

> Message DB is a good fit in a microservice-based architecture I don't know about that, I have seen many microservices that process data that comes in from messages on message queues; frequently that is a better design than receiving the data over synchronous http PUT or POST. By "message queues" I mean AWS SNS/SQS and Azure event hubs. Would a SQL Db be "a good fit in a microservice-based architecture" replacement…

Take a look at that 2nd link in the grandparent post. I remember it as one that was often cited when this debate first went around the Internet half a decade or so ago. The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed. One could argue that that is the real antipattern. It'll hurt you big tim…

> you're putting a key piece of smarts into the queue: Tracking whether a message has been processed.

yes, AWS SQS does that. e.g. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...

Are you saying that this is bad? It doesn't seem so.

> you're losing the ability to add a second listener without either affecting what messages are being seen by (and therefore the behavior of) the existing listener(s).But, if you're doing microservices, you're probably trying to keep things more flexible than that, and want to allow them to evolve more independently.

I'm not following. Both AWS SNS/SQS and Azure Event hubs were able to handle that scenario just fine, by design.

In AWS you attach multiple queues to the same SNS endpoint and have a pool of subscribers on each queue, in Azure you declare a "consumer group" and a pool of subscribers in it. Or declare a new consumer group if need be.

We regularly scaled up AWS from 3 listeners to 30 depending on load, and back down again, or replaced instances one by one, and still it guaranteed at-least once delivery (in practice, exactly once in the vast majority of cases) across the subscribers. We _never_ "lost the ability to add another listener" either scaling up in the same pool, or creating a new pool.

Re: Message DB: Event Store and Message Store for PostgreSQL

#65

Post by sbellware » Tue, Dec 17, 2019, 5:53 PM UTC

Thaxll wrote: post #46

sgk284 wrote: post #19

Earlier quoted context omitted.

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

Running a PG instance on DO for a startup is a really bad idea, I don't understand how can choose that over a managed solution. If you're a startup just use SNS / Kinesis / Google pub sub ect ...

Or managed Postgres on AWS Aurora, AWS RDS, Google Cloud SQL, Heroku, etc :)

SNS, Kenesis (Kafka), Google Pub/Sub are awesome. Not the same problem/solution fit as an event store, but awesome for the scenarios and architectures they're targeted at.

Re: Message DB: Event Store and Message Store for PostgreSQL

#66

Post by SideburnsOfDoom » Tue, Dec 17, 2019, 5:54 PM UTC

sbellware wrote: post #63

mumblemumble wrote: post #61

Earlier quoted context omitted.

Take a look at that 2nd link in the grandparent post. I remember it as one that was often cited when this debate first went around the Internet half a decade or so ago. The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed. One could argue that that is the real antipattern. It'll hurt you big tim…

> The almost-but-not-quite unstated major premise of the whole argument is that you're putting a key piece of smarts into the queue: Tracking whether a message has been processed. One could argue that that is the real antipattern. Indeed. Lots of good writing on the tradeoffs, for example: https://sookocheff.com/post/messaging/dissecting-sqs-fifo-qu... At the risk of beating the point to death, the transition from SO…

> the transition from SOA to Microservices is marked by the transition of smart pipes to dumb pipes, and dumb pipes have no knowledge of whether a message has been processed.

That doesn't to seem to be characteristic of Microservices at all. It's more about small bounded context and independent deployment of the service. All of which is completely possible over the "smart pipes" that message queues that I mentioned will give you.

Insisting that this wheel must be re-invented or worked around to make it a microservice seems very odd.

Re: Message DB: Event Store and Message Store for PostgreSQL

#67

Post by sbellware » Tue, Dec 17, 2019, 6:10 PM UTC

adverbly wrote: post #4

Looks cool. I can't find any performance or benchmarks though. Anyone care to guess/ballpark what kind of write speeds single nodes might be able to handle?

There are too many "it depends" issues, especially the hardware, and not least the application architecture sitting on top of it.

There's a script in the Message DB codebase that will do some fairly rudimentary read and write performance measures for your Postgres deployment:

https://github.com/message-db/message-db/blob/master/databas...

It's run right on the database, so no network I/O or client implementation in the loop - which is inevitably non-representative of "real world" scenarios.

Re: Message DB: Event Store and Message Store for PostgreSQL

#68

Post by frafra » Tue, Dec 17, 2019, 6:20 PM UTC

kiwicopple wrote: post #14

For anyone just looking for the pub/sub functionality, I have been developing something that provides the functionality for PostgreSQL: https://github.com/supabase/realtime It's an Elixir server (Phoenix) that allows you to listen to changes in your database via websockets. Basically the Phoenix server listens to PostgreSQL's replication functionality, converts the byte stream into JSON, and then broadcasts over webs…

Similar, using triggers, Python, aiohttp + asyncpg: https://github.com/frafra/postgresql2websocket/

Re: Message DB: Event Store and Message Store for PostgreSQL

#69

Post by ethangarofolo » Tue, Dec 17, 2019, 6:55 PM UTC

SideburnsOfDoom wrote: post #57

ethangarofolo wrote: post #50

Earlier quoted context omitted.

In addition to what the sibling comments to this one said, Message DB solves a different problem than what queues solve. Message DB is a good fit in a microservice-based architecture, and the "micro" in "microservice" comes from concept of Dumb Pipes / Smart Endpoints ( https://martinfowler.com/articles/microservices.html#SmartEn... ). Message DB is a "dumb pipe," whereas queues and brokers fit into the "smart pipe"…

> Message DB is a good fit in a microservice-based architecture I don't know about that, I have seen many microservices that process data that comes in from messages on message queues; frequently that is a better design than receiving the data over synchronous http PUT or POST. By "message queues" I mean AWS SNS/SQS and Azure event hubs. Would a SQL Db be "a good fit in a microservice-based architecture" replacement…

100% with you that messages are a better way for microservices to receive their input. I don't see how a service could receive its input over HTTP and still retain its autonomy.

A message store like Message DB can at the same time serve as record of system state and communication channel. Writing a message to the store is the same as publishing it for consumers to pick up. Messages are written to streams, and interested parties subscribe to those streams by polling them for new messages.

The store itself isn't aware of which components are subscribing or managing their read positions.

So to your question, the first action of a component receiving a message is do whatever it does to handle the message and then write new messages (specifically events) to record and signal what happened in response to the original message.

Re: Message DB: Event Store and Message Store for PostgreSQL

#70

Post by mike_hock » Tue, Dec 17, 2019, 6:56 PM UTC

sbellware wrote: post #62

SideburnsOfDoom wrote: post #59

Earlier quoted context omitted.

> Sooner or later, in anything built with a computer, there's going to be polling. ... But under the hood, there's polling. Is that true? I am reminded of this essay that goes into a deep dive of what happens under the hood with http and the underlying network I/O https://blog.stephencleary.com/2013/11/there-is-no-thread.ht... And at the lowest level, the network card interrupts the CPU because it has finished readin…

Anything that processes a signal checks if a signal has been received. It's no so black-and-white at the level of electricity, but higher-level things at the level of durable message queues check for new signals, even if those signals arrive via "push".

It is still good design to do the polling only at the lowest level where you must. Higher layers should be reactive.