Cool example. But, doing this will create a tight coupling between business logic , the state machine, and storage/persistence, Postgres. If ever you decide you want to migrate away from Postgres, you'll need to rewrite all the business logic into some other out of process language. If ever you want to scale this, such that you want to calculate states in batches, you'd want to have the business logic somewhere else.…
Author here. > If ever you decide you want to migrate away from Postgres, you'll need to rewrite all the business logic into some other out of process language. Yeah, but I don't consider this a bad thing. IMO migrating between databases should always require a lot of rewriting. If it doesn't, you're most likely underutilizing the features provided by your database. > If ever you want to scale this, such that you wan…
Implementing State Machines in PostgreSQL
71–80 of 90 posts
Re: Implementing State Machines in PostgreSQL
#72Cool trick. Should be an enum instead of text though. Also the transition table might be an actual SQL table as well instead of switch-cases in a function. - That way it remains a bit more declarative and you can do some meta-queries.
Yeah, maybe I'll update the post and mention ENUMs. I like the type safety of ENUMs, and I'm actually using them in my application, but they can cause a lot of problems. E.g. it's currently almost impossible to remove an ENUM value [1], and adding a new value can't be done inside of a transaction. The state transitions could definitely go into a table, but I feel it's overkill in many cases, and would have probably o…
I agree with your comments about ENUMs, though, especially having built many "static" products that suddenly need their ENUMs changed, and having a group of developers visiting me and asking why it's not as possible as they had assumed :)
Re: Implementing State Machines in PostgreSQL
#73Oh no... I commend your effort, but this is totally ill-conceived. What's even more disheartening is the number of people who looked at this and also thought it was a good idea. There is just so much wrong here... let's start with the basics. If this is truly a FSM then why on earth are you using a transaction table (state is a value)? An accumulating snapshot table (state is a field), besides being EXACTLY for this…
TBH I'm not sure if I follow your argument. You can certainly apply the main idea of this article (modeling a FSM as a user defined aggregate) to automatically materialize the latest state of each order. In fact, that's what we're doing in the app we're using this technique in. Anyway, YMMV and I'm not recommending to apply the ideas in this article in every situation. But after having had all of this logic in the ap…
Furthermore, I'm challenging the utility of putting the transaction logic in the database. It doesn't make sense other than to seem elegant. His application needs to actually respond to the events so it's just silly to separate the transitions. This is a complete redo.
Re: Implementing State Machines in PostgreSQL
#74Earlier quoted context omitted.
TBH I'm not sure if I follow your argument. You can certainly apply the main idea of this article (modeling a FSM as a user defined aggregate) to automatically materialize the latest state of each order. In fact, that's what we're doing in the app we're using this technique in. Anyway, YMMV and I'm not recommending to apply the ideas in this article in every situation. But after having had all of this logic in the ap…
What I'm getting at is that the author went ahead and implemented a bad solution to a simple problem because it seemed "cool". This is a great example of what NOT to do. A simple accumulating snapshot table with a bit field along with a timestamp for each state and some simple constraints would do the exact same thing more efficiently. Furthermore, I'm challenging the utility of putting the transaction logic in the d…
Re: Implementing State Machines in PostgreSQL
#75Earlier quoted context omitted.
I hear the argument about switching storage layers a lot, and I don't completely disagree with it, but in my 20+ years writing code, I've never found a case where switching storage engines didn't cause a massive rewrite even when the storage layer was used in an agnostic way. I'm sure there are examples where it has worked, but saying it as if it's a maxim just feels wrong to me.
Could you give some specifics as to what kind of stuff typically needs refactoring if you switch storage layers?
And even where there are no feature or syntax issues there may well be optimisation differences. For instance going from MSSQL to postgres you might hit a significant difference with CTEs because MSSQL can perform predicate optimisations through them but postgres doesn't - this might mean core queries need to be refactored significantly for performance reasons (to avoid extra index or table scans) if not functional ones. (not intending to pick on postgres here, I'm sure there are similar examples in the other direction and between other engines, but this is the most significant example that immediately springs to mind).
Re: Implementing State Machines in PostgreSQL
#76Re: Implementing State Machines in PostgreSQL
#77Earlier quoted context omitted.
There are good deployment tools for databases, e.g. ones for which all of the database 'code' objects (or just all of the objects period) are maintained in source control and updates are either automatic or scripted. Of course updating a database is much harder than overwriting executable or library files, but a lot of the objects in a database should be safely updatable by simply dropping and recreating them. The tr…
A great one is http://sqitch.org
My favorite, and the only one I've used extensively, is [DB Ghost](http://www.dbghost.com/). What I like about it compared to all others I've run across is that it, by default, will automatically sync your target DB (e.g. a production DB) with a model source DB that it also builds automatically.
So instead of scripting out every schema change as explicit SQL statements and queries you just maintain the scripts to build the model source DB, e.g. to add a column to a table, instead of creating a SQL script file to `ALTER TABLE Foo ADD COLUMN Bar ...` you just update the existing SQL script file with the `CREATE TABLE Foo ...` statement. When you deploy changes – 'sync' a target DB in the DB Ghost terminology – it automatically detects differences and modifies the target to match the source.
The benefit being that neither you nor the DB Ghost program needs to explicitly perform every single migration since the beginning of time. Only changes that need to be explicitly handled as migrations, but, with a little customization, doing that is pretty easy too.
The bar now for me working with databases is whether I can create a new 'empty' database (with test data) in a minute or two and whether I can automatically deploy changes (or, as I'm doing now, generate a deployment script automatically). Given that, I can actually do something like TDD for databases, which is really nice, especially if there's significant business logic in the database (which there almost always is in my experience to-date).
Re: Implementing State Machines in PostgreSQL
#78Earlier quoted context omitted.
My thoughts exactly. I don't do a ton of DB programming, but I've only ever written one thing in a non-agnostic way. We got a requirement that users wanted to copy an entire "project" which was the top level of a hierarchy. I wrote an Oracle routine to do the deep copy. I did it because I imagined what the PL/SQL would look like (clean) vs. what the Java code would look like (considering Hello World in Java is ugly,…
Mine too! Having logic embedded into SQL functions seems to be an anti-pattern to me (it's harder to maintain and harder to do release management). While it's great that Postgres can do this (btw, I love Postgres), I suspect there aren't many people will use this feature in production environment.
How is this any different than anything else where DB data & application logic have to be kept in sync?
version control + loader/change scripts should handle it, no?
Re: Implementing State Machines in PostgreSQL
#79Earlier quoted context omitted.
TBH I'm not sure if I follow your argument. You can certainly apply the main idea of this article (modeling a FSM as a user defined aggregate) to automatically materialize the latest state of each order. In fact, that's what we're doing in the app we're using this technique in. Anyway, YMMV and I'm not recommending to apply the ideas in this article in every situation. But after having had all of this logic in the ap…
What I'm getting at is that the author went ahead and implemented a bad solution to a simple problem because it seemed "cool". This is a great example of what NOT to do. A simple accumulating snapshot table with a bit field along with a timestamp for each state and some simple constraints would do the exact same thing more efficiently. Furthermore, I'm challenging the utility of putting the transaction logic in the d…
Re: Implementing State Machines in PostgreSQL
#80Earlier quoted context omitted.
What I'm getting at is that the author went ahead and implemented a bad solution to a simple problem because it seemed "cool". This is a great example of what NOT to do. A simple accumulating snapshot table with a bit field along with a timestamp for each state and some simple constraints would do the exact same thing more efficiently. Furthermore, I'm challenging the utility of putting the transaction logic in the d…
I don't understand your alternative solution, nor your negativity. Would you still keep the events table, or just have a single orders table with the latest state?
So let me explain in more detail how you could solve this without all of the overhead you have created. Assuming a true FSM, there is no need to store the state as a value in your database (e.g. "awaiting_payment" in a [state] field). Instead, you should be storing your states as BIT fields:
[awaiting_payment], [awaiting_shipment], [shipped], etc
where "1" indicates the current state and every other state field is "0". In addition, you will need a nullable timestamp for each state:
[awaiting_payment_timestamp], [awaiting_shipment_timestamp], etc
to store when each state was entered, if at all. In this way, every order is in a single row and your analytics become trivial queries that don't require any complex joins or correlated sub-queries. Additionally, using simple constraints on each state field (e.g. the [awaiting_shipment] field 1 if the [awaiting_payment_timestamp] is NULL) would remove the need for your trigger and, again, reduce overhead.
This is called an "accumulating snapshot" table, and it is used in data warehousing to solve the exact problem you are dealing with here - tracking an entity through a series of predetermined events to monitor it's status and mark milestones. Using a "transaction" table like in your example, while certainly possible, is storing your data in a way that you cannot easily answer the questions you may want to ask of it. You are spreading your entity across multiple rows instead of putting it all in one row.
Think about how the data translates to an object in your domain. Let's call this table "order_state" ("order_events" is a bad name for two reasons: it's plural - which is just wrong - and it doesn't faithfully describe your entity). I'm going to assume "order" and "order_item" tables as well. These three tables create an aggregate root that can be used to derive an Order object in your domain. This Order will have, at the very least, 3 properties corresponding to your entity's attributes in your database:
id, items, state
where "items" is a collection of OrderItem and state is OrderState (see where I'm going here?). The question becomes about how the OrderState is populated. If your Order object represents the "context" in your FSM, then the OrderState is the base class for all states:
AwaitingPaymentState, ShippingState, CancelledState, etc
This means polymorphism is inevitable. There are 3 common ways to deal with polymorphism in a database, but using a transaction table (like in your example) is not one of them. For your problem, utilizing "single table inheritance" (where all objects in a hierarchy map to a single table row) makes the most sense because you are not storing different data in each child class, rather, just a marker to indicate which state is current (this is also generally the most performant solution). It also means you don't need to add any additional tables, because your "order_state" table can serve both purposes.
Using a transaction table, you are adding a 3rd orthogonal dimension to your aggregate (another set of primary keys), so your object graph becomes needlessly large and it becomes a chore to choose and hydrate your current state.
My last critique is in regard to the choice of putting your transition logic in the database itself. I just don't see how that can be useful other than to "seem" clean. What I mean is, somehow an order actually has to be shipped or an item has to be paid for, and adding a row to a database table doesn't do that. So somewhere else in your stack you must have duplicated your transaction logic in some fashion in order to facilitate the correct database calls when the events take place in your domain.
For example, using my above objects, when a the "pay" event is raised on your Order object "context", it will invoke the "pay" event on it's current state, AwaitingPaymentState, to undergo transition. In this scenario, there must be some method (an "action") that actually handles the payment. Either there is no state machine at all in your domain and you need to validate the state before processing the payment, OR you have a state machine like my example and you need to transition to the AwaitingShippingState. In either case, the logic is duplicated.
What you have created is a cool academic solution that may work (I don't doubt that it works), but this problem was solved 30 years ago and if this was done on my team, I would insist it be redone to adhere to the basic principals of the entity relationship model and database design.
Does this make more sense?