Live data from Hacker News

Message DB: Event Store and Message Store for PostgreSQL

blog.eventide-project.org

91–100 of 104 posts

Re: Message DB: Event Store and Message Store for PostgreSQL

#91
post #76
post #71

Earlier quoted context omitted.

You hit on the real reason architects push microservices in smaller companies: it limits options to engineers and to the startup by putting up walls in the form of network rules that are resistant to change. A startup needs all the options on the table because the alternative is they go out of business and there is no startup. This idea that we can enforce “good architecture” (subject to interpretation) with technolo…

Services permit scaling development teams. In a startup where you're all in the same room, lots of separate services don't really make sense, and you probably don't know where to make the right cuts even if you tried. When you want to grow beyond the one big room, with a lot of teams, potentially in other time zones and countries, then you want to be able to carve off services so teams can own them.

The problem here is that an unmitigated monolith with no domain partitioning in its data model can't be transformed into a service architecture by carving off pieces.

For pieces to be able to be carved off, they already have to be autonomous.

What usually happens is that devs without experience in service architectures presume that service architectures are probably just the things they are used to seeing, ie: monolithic entities.

It's usually then that we hear things like "extract the product service". The problem is that product is an entity, and entities are the last thing to build services around. That's how we end up with distributed monoliths, and subsequently failed service architecture projects.

In order for an app to be able to transition to services, the app has to be designed this way from the beginning.

And yes, it's definitely possible to know what the model partitions should be up front. They're very natural divisions. But they can't be arrived at by looking through an entity-centric lens. And unfortunately, forms-over-data apps very rarely provide us with an opportunity to learn about the "other" way to do it.

Re: Message DB: Event Store and Message Store for PostgreSQL

#92
post #19

On behalf of a technology team, looking to incorporate a messaging platform to our landscape: I am curious to understand the motivation behind implementing a messaging system on top of a database technology. There are robust offerings both for small scale and large throughput systems. What are the benefits of this project over other implementations(RabbitMQ, Kafka, Celery, ActiveMQ, ZeroMQ, SNS/SQS)?

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

This! So many wasted years of engineering making correct applications on top of Cassandra/Kafka/Elasticsearch when a single Postgres server could have handled everything no problem.

Scalability for a tech product should be a concern at some point, but if you don’t have product market fit it doesn’t matter. The complexity of building a distributed system right out of the gate will cripple rapid iteration or lead to a broken product.

Re: Message DB: Event Store and Message Store for PostgreSQL

#93
post #19

Earlier quoted context omitted.

At my previous startup we put everything we could into the database (work queues and pup/sub included). It was magical. It minimized complexity (moving parts, boundaries, etc...), everything was transactional, and a single dump of the database gave you a copy of the entire persistent state of your system at that moment (across all services). We had several services running, but any notion of persistent state was stor…

For some perhaps-off-topic contrarianism, what do you make of Uncle Bob railing against database-oriented architectures? [0] (He takes a good few minutes to get his point across.) [0] https://youtu.be/o_TH-Y78tt4?t=2566

I think his take on databases is moronic with a dash of common sense:

He rails against a strawman where people put SQL everywhere, into views, into application logic ("mail merge" is one he mentions), and DBAs gatekeep everything.

The common sense part: no, your views shouldn't be composing SQL with string interpolation.

The moronic parts:

- He venerates application logic and regards the data model as a detail, but data models aren't details — they tend to outlive application logic.

- He holds up in-memory data structures as a platonic ideal, but doesn't address the things databases provide for you: schemas, constraints, transactional semantics and error recovery, a clear concurrency story.

Re: Message DB: Event Store and Message Store for PostgreSQL

#94

Earlier quoted context omitted.

I can't watch that at work right now. But is it pretty much this? https://blog.cleancoder.com/uncle-bob/2012/05/15/NODB.html I've heard of him before but never really read anything much by him. Is this guy for real? He worked at one startup, got angry about a database, became a consultant, and screams about everyone being wrong about everything?

Yes he makes similar points there. A few scattered critical thoughts: * I don't see any problem with stored procedures. They can make good sense for, say, auditing, as well as for performance. * People describe their software systems as "Using Oracle" because it matters, not because their design is stupid. It tells me that Oracle skills are relevant, for instance. * NoSQL is generally, in my experience, awful and cha…

Yeah, seems like he vastly overstates things. ssds are great for relational databases. They all use them.

Re: Message DB: Event Store and Message Store for PostgreSQL

#95
post #74
post #47

Earlier quoted context omitted.

You obviously need to write queries which span tables (I.e. joins). I can see how you misunderstood me there however, it was admittedly poorly worded. What I was talking about was basic data hygine as it's often called. If you need specific data, you define a clear way to get this data and only access it through that API. This can be a class, method or anything your language of choice prefers. If you skip that step a…

Having scaled up the tech in a company from $0 to $10+MM ARR, from zero customers to over 300 million records a day, I think you are simply incorrect and uninformed about the trade-offs that make sense in a startup.

so you are the only one with this knowledge, sensei?

Re: Message DB: Event Store and Message Store for PostgreSQL

#96

Earlier quoted context omitted.

> you're putting a key piece of smarts into the queue: Tracking whether a message has been processed. yes, AWS SQS does that. e.g. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS... Are you saying that this is bad? It doesn't seem so. > you're losing the ability to add a second listener without either affecting what messages are being seen by (and therefore the behavior of) the existing listener(s).But,…

> Are you saying that this is bad? It doesn't seem so. It's not "bad" per se, but it's not what it seems on the surface. ACKing a message doesn't mean that the message will not be received more than one time. As long as the smarts for recognizing recycled messages are embedded in the application logic, all will be fine. Here's a good explanation written about SQS, but applies to all message technologies that work bas…

> Here's a good explanation written about SQS, but applies to all message technologies that work based on ACKs

Deals with FIFO and ordering. You also mention "historical messages".

Sure, if those are things that you need, then you almost certainly want a different tech to SQS, likely Kafka. I won't disagree with that, except as in making it the focus of the criticism of SQS and similar message queues. It isn't that thing.

Re: Message DB: Event Store and Message Store for PostgreSQL

#97
Reach out to HYDRAHACKING01 @ HOTMAIL DOTCOM, for any type of hacking job to be done. Remove negative reports from your credit score, spy on your partner, deleted or encrypted messages and datas, to following up all social media activities anonymously, gaining full access to any mobile device, Hydra Hacking Inc are Prolific Professional Investigators, and they are one of the best groups (if not the best.) in the open market. i am a living witness

Re: Message DB: Event Store and Message Store for PostgreSQL

#98

Earlier quoted context omitted.

Yes he makes similar points there. A few scattered critical thoughts: * I don't see any problem with stored procedures. They can make good sense for, say, auditing, as well as for performance. * People describe their software systems as "Using Oracle" because it matters, not because their design is stupid. It tells me that Oracle skills are relevant, for instance. * NoSQL is generally, in my experience, awful and cha…

Yeah, seems like he vastly overstates things. ssds are great for relational databases. They all use them.

I think the point is that you might be able to build something else than a standard relational db and make it faster.

Something like VoltDB but there are other solutions. There used to be some blogs about their architecture and how they could avoid a couple of steps a normal RMDB needs that takes a lot of time.

https://www.voltdb.com/why-voltdb/

Re: Message DB: Event Store and Message Store for PostgreSQL

#99
post #68

For anyone just looking for the pub/sub functionality, I have been developing something that provides the functionality for PostgreSQL: https://github.com/supabase/realtime It's an Elixir server (Phoenix) that allows you to listen to changes in your database via websockets. Basically the Phoenix server listens to PostgreSQL's replication functionality, converts the byte stream into JSON, and then broadcasts over webs…

Similar, using triggers, Python, aiohttp + asyncpg: https://github.com/frafra/postgresql2websocket/

Any reason for your choice of asyncpg vs aiopg? Curious as I have worked with both

Re: Message DB: Event Store and Message Store for PostgreSQL

#100

Earlier quoted context omitted.

Yeah, seems like he vastly overstates things. ssds are great for relational databases. They all use them.

I think the point is that you might be able to build something else than a standard relational db and make it faster. Something like VoltDB but there are other solutions. There used to be some blogs about their architecture and how they could avoid a couple of steps a normal RMDB needs that takes a lot of time. https://www.voltdb.com/why-voltdb/

> I think the point is that you might be able to build something else than a standard relational db and make it faster.

I sincerely doubt that.

For it to be useful in real world situations, you'd have to build your own highly flexible, scalable, rock-solid, high-performance data-management solution, ideally one which enables the user to use a declarative means of expressing queries.

This is, of course, a DBMS.

If you build one which imposes structure on the data, you've got either a relational DBMS, or an object DBMS, or some other kind of well-studied database solution... except you've rolled your own from scratch, without input from database experts, so it's going to be a disaster.

Unless you're someone like Microsoft, Google, or Amazon, you pretty much can't build one of those. It costs tens of millions. It takes a huge amount of testing, for obvious reasons.

I really don't see any argument for not using a mature database system for managing a typical company's data.

Of course, if we aren't talking about a full-scale database that has to cope with a huge amount of messy mission-critical data in a changing business environment, then the game changes, and sure, you might have a chance just writing something yourself.

Netflix's video-streaming CDN ('OpenConnect'), for instance, obviously isn't powered by an RDMS. They put in a huge amount of highly technical work building their own finely tuned technologies to pull data off the disk and get it to the NIC with minimal glue in between. But that's Netflix, and virtually no real companies face that kind of challenge.

Also, VoltDB looks like a streaming DB technology. Isn't that both an DBMS (rather than a means of rolling-your-own), and very niche? I don't see how it's relevant here. If it's really able to serve business needs better, then great, but it's still a complex-DBMS-as-a-product.

Post reply on HN