Message DB: Event Store and Message Store for PostgreSQL • Hacker News Board

Re: Message DB: Event Store and Message Store for PostgreSQL

#71

Post by nthj » Tue, Dec 17, 2019, 7:01 PM UTC

barrkel wrote: post #37

y4mi wrote: post #33

Earlier quoted context omitted.

That's incorrect. It's trivial If you have a clearly defined API with which to extract or insert data. It only starts to leak if everyone accesses the the DB through raw sql, mishmashing modules and creating insane dependencies. Give each module ownership to a table and demand to only access it through there and the backend storage system becomes irrelevant

If you put everything in the database, and use it for all it's able to do, as the comment I replied to suggested, it is practically inevitable under startup feature pressure conditions that total modularity won't hold. And in fact I think it would be irresponsible to pursue total modularity; it would court failure. I believe you're being deeply naive about modules owning single tables. You forego relational integrity…

You hit on the real reason architects push microservices in smaller companies: it limits options to engineers and to the startup by putting up walls in the form of network rules that are resistant to change.

A startup needs all the options on the table because the alternative is they go out of business and there is no startup. This idea that we can enforce “good architecture” (subject to interpretation) with technology by isolating junior engineers with networking rules needs to at least be more transparent about its motivations.

Re: Message DB: Event Store and Message Store for PostgreSQL

#72

Post by SideburnsOfDoom » Tue, Dec 17, 2019, 7:06 PM UTC

mike_hock wrote: post #70

sbellware wrote: post #62

Earlier quoted context omitted.

Anything that processes a signal checks if a signal has been received. It's no so black-and-white at the level of electricity, but higher-level things at the level of durable message queues check for new signals, even if those signals arrive via "push".

It is still good design to do the polling only at the lowest level where you must. Higher layers should be reactive.

Does a network card "poll" ? it's hardware activated by current flowing into it. Does the CPU poll the card, no, the CPU is interrupted by the network card, again by receiving an electrical signal.

If there's polling, it happens in a matter of a few CPU cycles.

Re: Message DB: Event Store and Message Store for PostgreSQL

#73

Post by mike_hock » Tue, Dec 17, 2019, 7:18 PM UTC

SideburnsOfDoom wrote: post #72

mike_hock wrote: post #70

Earlier quoted context omitted.

It is still good design to do the polling only at the lowest level where you must. Higher layers should be reactive.

Does a network card "poll" ? it's hardware activated by current flowing into it. Does the CPU poll the card, no, the CPU is interrupted by the network card, again by receiving an electrical signal. If there's polling, it happens in a matter of a few CPU cycles.

At some point, this is splitting hairs. Single instructions are atomic wrt. interrupts, so surely there must be some sort of check every cycle whether an interrupt has arrived during that cycle.

The magnitude of the time slice or polling interval is immaterial as to whether it is to be considered "polling."

Re: Message DB: Event Store and Message Store for PostgreSQL

#74

Post by barrkel » Tue, Dec 17, 2019, 7:35 PM UTC

y4mi wrote: post #47

barrkel wrote: post #37

Earlier quoted context omitted.

If you put everything in the database, and use it for all it's able to do, as the comment I replied to suggested, it is practically inevitable under startup feature pressure conditions that total modularity won't hold. And in fact I think it would be irresponsible to pursue total modularity; it would court failure. I believe you're being deeply naive about modules owning single tables. You forego relational integrity…

You obviously need to write queries which span tables (I.e. joins). I can see how you misunderstood me there however, it was admittedly poorly worded. What I was talking about was basic data hygine as it's often called. If you need specific data, you define a clear way to get this data and only access it through that API. This can be a class, method or anything your language of choice prefers. If you skip that step a…

Having scaled up the tech in a company from $0 to $10+MM ARR, from zero customers to over 300 million records a day, I think you are simply incorrect and uninformed about the trade-offs that make sense in a startup.

Re: Message DB: Event Store and Message Store for PostgreSQL

#75

Post by bobbydreamer » Tue, Dec 17, 2019, 7:37 PM UTC

kiwicopple wrote: post #30

hashhar wrote: post #26

Earlier quoted context omitted.

Look at embedded Debezium. https://debezium.io/docs/embedded/ Very similar to what you are doing but instead of Websockets you can decide where to send your data. The core-debezium sends it to Kafka.

I took a lot of inspiration from Debezium - at the time they required the wal2json plugin (not sure if that's still the case?). I didn't have an option to install the plugin and I mostly wanted a websocket-friendly implementation to replace Firebase. And if I'm 100% honest I just wanted to make something i thought was cool

Why replace Firebase ?

Re: Message DB: Event Store and Message Store for PostgreSQL

#76

Post by barrkel » Tue, Dec 17, 2019, 7:37 PM UTC

nthj wrote: post #71

barrkel wrote: post #37

Earlier quoted context omitted.

If you put everything in the database, and use it for all it's able to do, as the comment I replied to suggested, it is practically inevitable under startup feature pressure conditions that total modularity won't hold. And in fact I think it would be irresponsible to pursue total modularity; it would court failure. I believe you're being deeply naive about modules owning single tables. You forego relational integrity…

You hit on the real reason architects push microservices in smaller companies: it limits options to engineers and to the startup by putting up walls in the form of network rules that are resistant to change. A startup needs all the options on the table because the alternative is they go out of business and there is no startup. This idea that we can enforce “good architecture” (subject to interpretation) with technolo…

Services permit scaling development teams. In a startup where you're all in the same room, lots of separate services don't really make sense, and you probably don't know where to make the right cuts even if you tried. When you want to grow beyond the one big room, with a lot of teams, potentially in other time zones and countries, then you want to be able to carve off services so teams can own them.

Re: Message DB: Event Store and Message Store for PostgreSQL

#77

Post by sbellware » Tue, Dec 17, 2019, 7:59 PM UTC

SideburnsOfDoom wrote: post #60

sbellware wrote: post #58

Earlier quoted context omitted.

Yes, we disagree :) The response given by @ethangarofolo does a good job of addressing the main points. I would say that the slide deck linked is a bit sneaky. Sooner or later, in anything built with a computer, there's going to be polling. Higher-level libraries abstract the polling so that it appears as push (rather than pull) to the application developer. But under the hood, there's polling. It should also be poin…

> persistent messaging is built on a database. Even RabbitMQ's persistent messaging is built on a database (Mnesia). Good point that any system that is "persistent" has a data store by definition. The real question is that if it's a good fit to a SQL relational database.

The right store is the one that has the features needed to implement the targeted patterns, has client libraries for most programming languages, can be tuned and scaled, has a large ecosystem, has numerous managed hosting solutions offered by popular cloud providers, and is approachable by the largest segment of the potential audience.

Postgres checks those boxes, as do others.

Technically, though, Postgres has been an "object-relational" database for some time.

In the end, Message DB uses a table as an append-only log, and leverages Postgres indexes, advisory locks, and JSON documents (and indexes) to implement some of the critical messaging features and patterns. It's not using the "relational" aspects of Postgres. There are no relational tables.

Here's the table schema: https://github.com/message-db/message-db/blob/master/databas...

Given the paucity of Postgres features that the message store leverages, it could have been implemented against the raw Postgres storage engine. Had we done that, though, few people could have understood it and been able to specialize it to their purposes using plain old SQL. And any performance improvements induced by skipping Postgres's tabular data abstractions would have been so negligible for the schema in question that it wouldn't have offered much return on the effort.

Re: Message DB: Event Store and Message Store for PostgreSQL

#78

Post by ziftface » Tue, Dec 17, 2019, 8:33 PM UTC

sbellware wrote: post #65

Thaxll wrote: post #46

Earlier quoted context omitted.

Running a PG instance on DO for a startup is a really bad idea, I don't understand how can choose that over a managed solution. If you're a startup just use SNS / Kinesis / Google pub sub ect ...

Or managed Postgres on AWS Aurora, AWS RDS, Google Cloud SQL, Heroku, etc :) SNS, Kenesis (Kafka), Google Pub/Sub are awesome. Not the same problem/solution fit as an event store, but awesome for the scenarios and architectures they're targeted at.

Is there a difference from a programming perspective between, say Kafka, and the event store linked in this post?

I understand that Kafka can scale horizontally and can handle crazy throughput, but I mean from a programming point of view, the idea of a unified log as a data model applies to both, correct?

Re: Message DB: Event Store and Message Store for PostgreSQL

#79

Post by pythonaut_16 » Tue, Dec 17, 2019, 9:16 PM UTC

kiwicopple wrote: post #14

For anyone just looking for the pub/sub functionality, I have been developing something that provides the functionality for PostgreSQL: https://github.com/supabase/realtime It's an Elixir server (Phoenix) that allows you to listen to changes in your database via websockets. Basically the Phoenix server listens to PostgreSQL's replication functionality, converts the byte stream into JSON, and then broadcasts over webs…

This is pretty cool.

Mnesia (one of the databases built into Erlang) has a similar ability to subscribe to its tables that I've used for realtime updates in some demo projects.

Re: Message DB: Event Store and Message Store for PostgreSQL

#80

Post by MaxBarraclough » Tue, Dec 17, 2019, 9:25 PM UTC

ianamartin wrote: post #55

MaxBarraclough wrote: post #39

Earlier quoted context omitted.

For some perhaps-off-topic contrarianism, what do you make of Uncle Bob railing against database-oriented architectures? [0] (He takes a good few minutes to get his point across.) [0] https://youtu.be/o_TH-Y78tt4?t=2566

I can't watch that at work right now. But is it pretty much this? https://blog.cleancoder.com/uncle-bob/2012/05/15/NODB.html I've heard of him before but never really read anything much by him. Is this guy for real? He worked at one startup, got angry about a database, became a consultant, and screams about everyone being wrong about everything?

Yes he makes similar points there. A few scattered critical thoughts:

* I don't see any problem with stored procedures. They can make good sense for, say, auditing, as well as for performance.

* People describe their software systems as "Using Oracle" because it matters, not because their design is stupid. It tells me that Oracle skills are relevant, for instance.

* NoSQL is generally, in my experience, awful and chaotic, rather than liberating. Lots of good work has gone into making serious grown-up relational databases. Schemas, normal forms, constraints, rigorous work on the ACID properties. Something like MongoDB is just sloppy amateur-hour by comparison.

* In the YouTube video, he says that solid-state drives render relational databases obsolete. This strikes me as absurd. Not even ultra-fast SSD storage technologies will do that. The relational model is effective, and the associated DBMSs still well justified. Joins belong in a query language, not in imperative code. Why would you want to try managing a huge complex dataset manually?

[0] https://en.wikipedia.org/wiki/3D_XPoint

(Edit: formatting)