Consistency is Consistently Undervalued • Hacker News Board

Re: Consistency is Consistently Undervalued

#111

Post by appleflaxen » Sat, Sep 17, 2016, 10:18 PM UTC

vidarh wrote: post #58

kahirsch wrote: post #48

Earlier quoted context omitted.

As you say, the low-level operations in banking do provide atomicity, which are often lacking in distributed databases. Also, the banking system does have the system of compensating transactions (e.g. returned checks) when constraints are violated, as well as an extensive system of auditing to maintain consistency. Do most distributed systems have the ability to detect inconsistency? How do they manage to correct pro…

> Do most distributed systems have the ability to detect inconsistency? They do if you build them to have them. Your bank account works without requiring consistency because every operation is commutative: You never do anything that says "set account to value x". You have operations like "deposit x" and "withdraw x". For operations that require you move money from one account to another, you need to guarantee that bo…

Does commutative mean something different in CS than in math? In math, commutative means that the order of operands doesn't matter.

For example: addition is commutative, because 1+2 = 2+1

Subtraction, however, isn't.

Since a transfer is basically a transfer from one account to the other, it couldn't possibly be commutative in a mathematical sense.

I suspect there might be a better term for the property you are trying to name.

"Atomic" seems adequate to me...

Re: Consistency is Consistently Undervalued

#112

Post by dunham » Sat, Sep 17, 2016, 11:10 PM UTC

appleflaxen wrote: post #111

vidarh wrote: post #58

Earlier quoted context omitted.

> Do most distributed systems have the ability to detect inconsistency? They do if you build them to have them. Your bank account works without requiring consistency because every operation is commutative: You never do anything that says "set account to value x". You have operations like "deposit x" and "withdraw x". For operations that require you move money from one account to another, you need to guarantee that bo…

Does commutative mean something different in CS than in math? In math, commutative means that the order of operands doesn't matter. For example: addition is commutative, because 1+2 = 2+1 Subtraction, however, isn't. Since a transfer is basically a transfer from one account to the other, it couldn't possibly be commutative in a mathematical sense. I suspect there might be a better term for the property you are trying…

(TL;DR - this is a concept that computer science borrowed from math)

In math (say abstract algebra / group theory), the operations on your database form a group under composition. Given two operations f,g (say f is "subtract $10" and g is "subtract $20"), the composition of those operations could be written "f•g" (depending on notation this can mean g is done first or f is done first, typically the former convention is used).

(So, instead of numbers and plus (+) in your example, we're using functions and composition (•) as our group.)

This group, of operations under composition, is said to be commutative if f•g = g•f for all f and g. (That is for all combinations of your possible operations.)

For addition/subtraction operations you're good to go.

If you can consider operations on a hash table being things like 'set a value of 10 for key "a"' the order that the operations are executed in do matter, and the group of those operations, under composition, is not commutative.

The subject of CRDT's in computer science is an attempt to create data types whose operations are commutative.

One reason to want do deal with commutative operations is that you don't have to figure out the order in which the operations officially occurred.

Re: Consistency is Consistently Undervalued

#113

Post by partisan » Sun, Sep 18, 2016, 3:02 AM UTC

victorNicollet wrote: post #36

partisan wrote: post #29

Earlier quoted context omitted.

My teams' event sourcing implementation stores and publishes events in a well defined order within a specific aggregate. There are scenarios where a two-phase commit is used to ensure that invariants across aggregates are maintained. We use an ACID compliant database to store our domain events and we project our events into a relational schema. When projecting events, we use transactions to make sure the database upd…

Yes, this is a typical way of doing things in the CQRS/ES world, but there are many things left unsaid: - would you have a single aggregate for "all bank accounts" so that one financial transaction equals one event, or is there a good reason to have multiple aggregates ? - is command execution transactional, or is it possible for another thread to generate (possibly conflicting) events between the time you check a pr…

I would have one aggregate instance per bank account. Each has a separate state and business rules would need to be applied at the individual account instance (e.g. overdraft fees, interest, etc). If you have to enforce rules at the bank level based upon actions taken within the account then you can have a bank aggregate that you can interact with through a saga and confirm changes through a two phase commit.

Command execution is not transactional, but writing to the domain event store is. That means that two writers cannot write the same domain event version number to the event store. Say an aggregate is at version 20. The next action would put it at 21. If two threads generate version 21 of the aggregate, only one of them can write it to the event store and the other gets an exception. Domain events are written in a transaction batch to the event store so we cannot get any writes in the middle. So, the first to write the next version wins. The other one loses.

We also partition our command and event processors by aggregate id. The same aggregate cannot have its events or commands processed more than one at a time because the processors are synchronous per partition.

If I understand you correctly then you have one instance of a very large aggregate per bounded context? If so, then no, it is not in the spirit of CQRS/ES. If you are handling commands in transactions, then the designers of your system may have chosen to favor consistency instead of the availability that CQRS provides.

Re: Consistency is Consistently Undervalued

#114

Post by wpietri » Sun, Sep 18, 2016, 6:51 AM UTC

betenoire wrote: post #109

knucklesandwich wrote: post #95

Earlier quoted context omitted.

There are definitely solutions for handling this with consistency semantics that are something weaker than linearizable consistency. But they usually become expensive in other ways and loose a lot of the availability benefits to recover the consistency properties you do need. Most of the models we're even talking about here are something in the realm of sequential or causal consistency, which are still considered str…

I'm seeing a lot of comments that are basically saying, "we don't need no stinking transactions because we can mostly reinvent them in the app." And the point of the article was, yeah, or you could pick a solution that already solves this.

Was that the point of the article? I didn't read it that way. E.g.: "In conclusion, if you use micro-services or NoSQL databases, do your homework! Choose appropriate boundaries for your systems."

Re: Consistency is Consistently Undervalued

#115

Post by victorNicollet » Sun, Sep 18, 2016, 7:32 AM UTC

partisan wrote: post #113

victorNicollet wrote: post #36

Earlier quoted context omitted.

Yes, this is a typical way of doing things in the CQRS/ES world, but there are many things left unsaid: - would you have a single aggregate for "all bank accounts" so that one financial transaction equals one event, or is there a good reason to have multiple aggregates ? - is command execution transactional, or is it possible for another thread to generate (possibly conflicting) events between the time you check a pr…

I would have one aggregate instance per bank account. Each has a separate state and business rules would need to be applied at the individual account instance (e.g. overdraft fees, interest, etc). If you have to enforce rules at the bank level based upon actions taken within the account then you can have a bank aggregate that you can interact with through a saga and confirm changes through a two phase commit. Command…

We do have the same constraint on event store writes, which we then use to implement command-level transactions : a command is a pure function that maps a command model to a set of events, and if the set of events cannot be written to the stream, the command model is updated and the function is called again. This does, indeed, have a very strong bias towards consistency, although the main objective is to allow multiple command processors per aggregate, which lets us embed the command processors in services that are inherently multi-instance (e.g. web applications).

I am not sure I understand why having multiple aggregates would provide availability. Unless the idea is that the events from different aggregates are stored on separate servers, so that a single-machine outage would only take down some aggregates ? If so, I agree that is the case in theory, though with limited practical applicability in our situation.

Re: Consistency is Consistently Undervalued

#116

Post by matthewmacleod » Sun, Sep 18, 2016, 12:45 PM UTC

bullen wrote: post #83

matthewmacleod wrote: post #13

Earlier quoted context omitted.

I don't mean to be unkind, but is this meant to be a parody? There are no "parts of a transaction" because a transaction is definitionally atomic. Transactional file modification is a fairly tricky problem, and I'd be surprised if you'd actually implemented a safe system in that manner. What's certain though is that spinning up MySQL or Postgres and using it to store simple records is essentially a zero-cost setup ta…

How is it atomic? this goes for all 5 of the replies (one of you did not downvote discussion, I'm guessing that person is the only one not from the US). How does mysql unwrite the deposit? There are always "parts" to everything. Two phase commit does not solve anything unless you have "undo the thing I did before the power was lost". Please refer to source code to prove your argumentation.

Please refer to source code to prove your argumentation.

I'm not digging through source code for you; transactional atomicity is a well-known and thoroughly researched problem.

Transactions in an ACID system are atomic by definition. They're designed in such a way that either an entire transaction occurs, or no part of a transaction occurs – in other words, it's not possible to partially apply a transaction, by design.

There are a bunch of different approaches to implementing this. I'd guess that MySQL does something like write the complete set of modifications in a transaction to disk as a separate buffer, and only once the entire transaction is complete updating some associated metadata to add the transaction to the database. A power failure at any point will result in an uncommitted transaction, which will have no effect on the database.

Here's the appropriate Wikipedia article for further reading -https://en.wikipedia.org/wiki/Atomicity_(database_systems) - lets suffice to say that if you are rejecting the idea that atomicity exists then I don't know what else to tell you. It's a core concept in information systems.

Re: Consistency is Consistently Undervalued

#117

Post by phamilton » Sun, Sep 18, 2016, 3:53 PM UTC

kpmah wrote: post #81

phamilton wrote: post #73

This profile example is missing the better approach: avoid the dependency of creating the user before creating the profile. Create the profile with a generated uuid. Once that succeeds, then create the user with the same uuid. If you build a system that allows orphaned profiles (by just ignoring them) then you avoid the need to deal with potentially missing profiles. This is essentially implementing MVCC. Write all y…

Well, we're playing make believe with the requirements. In fairyland the user and the profile need to exist together. Only the fairies know why! If you can relax the requirements, you can relax the constraints.

That's the point. Requirements are rarely as strict as we presume they are.

Re: Consistency is Consistently Undervalued

#118

Post by partisan » Sun, Sep 18, 2016, 3:59 PM UTC

victorNicollet wrote: post #115

partisan wrote: post #113

Earlier quoted context omitted.

I would have one aggregate instance per bank account. Each has a separate state and business rules would need to be applied at the individual account instance (e.g. overdraft fees, interest, etc). If you have to enforce rules at the bank level based upon actions taken within the account then you can have a bank aggregate that you can interact with through a saga and confirm changes through a two phase commit. Command…

We do have the same constraint on event store writes, which we then use to implement command-level transactions : a command is a pure function that maps a command model to a set of events, and if the set of events cannot be written to the stream, the command model is updated and the function is called again. This does, indeed, have a very strong bias towards consistency, although the main objective is to allow multip…

Aggregates represent consistency boundaries. To enforce consistency you would theoretically need to have your entire state loaded when you go to process a command for a single bank aggregate.

Since each command produces a new version of the bank and many commands are coming in at the same time, most of them will fail when writing to the event store. Do they keep retrying until they succeed? Either way, this is not efficient and could effect the availability of your servers.

If you have multiple aggregates, the consistency boundaries are smaller and therefore you have a smaller state that you have to maintain consistency across. There are less operations on a single aggregate and less opportunities for contention that arise from simultaneous operations.

Also, if you are using queues for your commands and events (we are, but I suspect you are not), then you can partition your queues such that you can process your workload N ways without worrying about things happening out of order. Each aggregate processes serially within a partition. If you have just one large aggregate then you would have to jump through a lot of hoops to be able to partition the work while maintaining ordering guarantees.

I can really only guess at the details of your implementation, but I am guessing that your design has accounted for all of the above in some way. If there is one thing I've learned, it's that there are many, many ways to implement CQRS. If your software works correctly under load, then I am not sure it would matter if it fits squarely into the definition of CQRS. In fact, you may have something new altogether that presents a better way to achieve the same goals as CQRS without the same cognitive overhead.

Re: Consistency is Consistently Undervalued

#119

Post by betenoire » Sun, Sep 18, 2016, 4:34 PM UTC

wpietri wrote: post #114

betenoire wrote: post #109

Earlier quoted context omitted.

I'm seeing a lot of comments that are basically saying, "we don't need no stinking transactions because we can mostly reinvent them in the app." And the point of the article was, yeah, or you could pick a solution that already solves this.

Was that the point of the article? I didn't read it that way. E.g.: "In conclusion, if you use micro-services or NoSQL databases, do your homework! Choose appropriate boundaries for your systems."

You are right. The article says to understand your choice, it doesn't say WHAT to choose.

Pretty sure the subtext is that developers are making their lives harder than they need to by not understanding the choice they are making.

Re: Consistency is Consistently Undervalued

#120

Post by bullen » Sun, Sep 18, 2016, 8:31 PM UTC

matthewmacleod wrote: post #116

bullen wrote: post #83

Earlier quoted context omitted.

How is it atomic? this goes for all 5 of the replies (one of you did not downvote discussion, I'm guessing that person is the only one not from the US). How does mysql unwrite the deposit? There are always "parts" to everything. Two phase commit does not solve anything unless you have "undo the thing I did before the power was lost". Please refer to source code to prove your argumentation.

Please refer to source code to prove your argumentation. I'm not digging through source code for you; transactional atomicity is a well-known and thoroughly researched problem. Transactions in an ACID system are atomic by definition. They're designed in such a way that either an entire transaction occurs, or no part of a transaction occurs – in other words, it's not possible to partially apply a transaction, by desig…

I don't think so; if a MySQL transaction has two separate tables, at some point it has to write one and then the other remembering to remove the first if the second fails with power outage for example.

I'm not rejecting anything, all I'm asking is; where is the code and how does it work? If you don't know then how come you are so confident it does work?

I'm pretty sure the "write some status to a file and rollback if transaction incomplete on startup" code is pretty horrible on all SQL systems. And on top of that it doesn't scale. Defending status quo is always worth questioning.