Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Consistency is Consistently Undervalued

kevinmahoney.co.uk

Post reply on HN
Print view View original

121–130 of 131 posts

Re: Consistency is Consistently Undervalued

#121

Post by tscs37 »

bunderbunder wrote: post #40
tscs37 wrote: post #31

Earlier quoted context omitted.

Totally agree. I've wet my toes with NoSQL systems but things like MongoDB just look like trouble waiting to happen. On the other hand, SQL makes it hard to maintain the state definition, I had to develop a mechanism to upgrade the DB from any possible state to the latest state. Still, this allows me to define very accurately what is a valid row or entry in the database, which means my Applications need to make far f…

It's absolutely true that migrations in SQL databases can be a pain. I don't actually think that's a bad thing - what's really going on there is that the DBMS is throwing data integrity issues into your face, and encouraging you to think about them. A lot of developers don't want to do that, because they want to think of data integrity as this tangential concern that's largely a distraction from their real job. A lot…

>DBMS is throwing data integrity issues into your face, and encouraging you to think about them.

And I love it to death for it.

It also allows me to do some nifty little tricks.

I've developed my own Migration Tool that works by exploiting this data integrity, it actually aids me to identify and upgrade the database dynamically without worrying about which updates are applied and which not.

Re: Consistency is Consistently Undervalued

#122

Post by victorNicollet »

partisan wrote: post #118
victorNicollet wrote: post #115

Earlier quoted context omitted.

We do have the same constraint on event store writes, which we then use to implement command-level transactions : a command is a pure function that maps a command model to a set of events, and if the set of events cannot be written to the stream, the command model is updated and the function is called again. This does, indeed, have a very strong bias towards consistency, although the main objective is to allow multip…

Aggregates represent consistency boundaries. To enforce consistency you would theoretically need to have your entire state loaded when you go to process a command for a single bank aggregate. Since each command produces a new version of the bank and many commands are coming in at the same time, most of them will fail when writing to the event store. Do they keep retrying until they succeed? Either way, this is not ef…

I understand better now what you're saying, though I would have called it "ability to scale" rather than "availability".

Our architecture prevents us from parallelizing the execution of commands within a bounded context. The ability to execute commands on any number of servers is for availability (transparent failover if an instance dies) rather than performance, since the event stream acts as a global lock anyway.

In practice, our system clocks in at a comfortable 1000 commands per second under stress tests ; during our peak hours and on our busiest aggregate, we only have to execute one command per second, so we can afford at least a x100 increase before we need to consider changing our architecture (and we're no longer an early-stage start-up, so x100 would mean a lot of revenue).

Similarly, the full state for our largest aggregate (not the command model, but the entire state of all projections) fits snugly in a few hundreds of megabytes, so all instances can keep it in-memory.

Re: Consistency is Consistently Undervalued

#123

Post by tscs37 »

pjungwir wrote: post #65
bunderbunder wrote: post #40

Earlier quoted context omitted.

It's absolutely true that migrations in SQL databases can be a pain. I don't actually think that's a bad thing - what's really going on there is that the DBMS is throwing data integrity issues into your face, and encouraging you to think about them. A lot of developers don't want to do that, because they want to think of data integrity as this tangential concern that's largely a distraction from their real job. A lot…

I am very fond of pushing as much validation into the database as I can, so I use `NOT NULL`, foreign keys, `CHECK` constraints, and whatever I can to let the database do the job of ensuring valid data. All of us make mistakes, and making sure there are no exceptions is just what computers are good at. I think you're right about "a lot of developers", but I find data modeling, including finding the invariants, pretty…

IMO validating in the DB also makes code faster.

When I get data from a well validated DB, I don't have to check that it is valid or has certain formats, that is ensured by the DB.

I can remove checks from the entire CRUD stack, if it's invalid data it does not get created or won't make it into the update.

The easiest way to check data is then to try the INSERT or UDPATE and if it works it's valid.

Otherwise I'd need to validate all fields, see if any contain errors, if any are missing, etc etc.

Re: Consistency is Consistently Undervalued

#124

Post by vidarh »

knucklesandwich wrote: post #82
vidarh wrote: post #68

Earlier quoted context omitted.

> You can't do this in general, because we often want to build applications where the user expects their actions won't be reordered. The key being they don't expect their actions to be reordered. In most systems this is trivially handled by logging event time as seen from the users system and ensure the user sees a consistent view of their own operations . There are exceptions, naturally, but the point is not that yo…

> Your privacy settings is a local concern to your account, it most certainly does not need to be globally consistent. All you need to do to avoid problems with that is to ensure consistency in routing requests for a given session, and enforce local consistency of ordering within a given user account or session depending on requirements. I think you're misunderstanding what the problem is here. I want global consiste…

> It's necessary that my own actions appear in their original order to me, but also to anyone else who is plausibly affected by their reordering.

No, it's necessary that the effects appear in the right order, and in the example case what matters is that the privacy setting is correctly applied to the subsequent post and stored with it.

> The full example also involves posting on a mutual friend's wall and having that post hidden by unfriending

If that is a requirement - and it's not clear it would be a good user experience, as it would break conversations (e.g. any reply; and what do you do about quoted parts of a mssage) and withdraw conversation elements that have already been put out there -, but in any case this does not generally require global consistency. It does require prompt updates.

Re: Consistency is Consistently Undervalued

#125

Post by vidarh »

uryga wrote: post #94
vidarh wrote: post #58

Earlier quoted context omitted.

> Do most distributed systems have the ability to detect inconsistency? They do if you build them to have them. Your bank account works without requiring consistency because every operation is commutative: You never do anything that says "set account to value x". You have operations like "deposit x" and "withdraw x". For operations that require you move money from one account to another, you need to guarantee that bo…

Correct me if I'm wrong, but I feel like bank transactions aren't always commutative. Consider: balance = 0 balance += 100 balance -= 50 You can't swap the order of addition and subtraction, because that would mean withdrawing from an empty account, and afaik you can't do that. Is there something I'm missing?

You can execute a withdrawal operation on an empty bank account. E.g. a credit card or a loan account are both accounts that are allowed to go negative, but regular accounts with overdrafts also do, and regular accounts without overdrafts are also often allowed to go negative (but often with prohibitive fees and restrictions attached).

It is entirely up to the bank whether or not it will be allowed to complete, but regardless the response from the bank the end result of the operations will be correct, but it may be different (it will be different if the bank will reject an operation in one order, but accept all of them when presented in a different order).

You are right that they are therefore technically not strictly commutative in the sense the outcome may differ in edge cases.

But the point is that what will not happen, is that you will get incorrect results. E.g. money won't "disappear" or "appear" spontaneously.

Re: Consistency is Consistently Undervalued

#126

Post by vidarh »

appleflaxen wrote: post #111
vidarh wrote: post #58

Earlier quoted context omitted.

> Do most distributed systems have the ability to detect inconsistency? They do if you build them to have them. Your bank account works without requiring consistency because every operation is commutative: You never do anything that says "set account to value x". You have operations like "deposit x" and "withdraw x". For operations that require you move money from one account to another, you need to guarantee that bo…

Does commutative mean something different in CS than in math? In math, commutative means that the order of operands doesn't matter. For example: addition is commutative, because 1+2 = 2+1 Subtraction, however, isn't. Since a transfer is basically a transfer from one account to the other, it couldn't possibly be commutative in a mathematical sense. I suspect there might be a better term for the property you are trying…

A bank transfer is rarely atomic unless within the same bank (and even then it depends how centralised their operations are). A transfer decomposes into operations on two separate accounts, which may involve additional steps (e.g. the sending bank will record having received a request to transfer it, record it's taken money out of the source account, record settlement, record the transfer has completed, etc.)

When talking about the operations being commutative, we're generally talking about each individual account, though it may apply wider as well.

In any case, the difference is that we are not talking about the elements of a given operation. You are right that if you see the operations as being "balance = balance + x" or "balance = balance - y", then the subtraction in the latter is not commutative.

But the operations seen as indivisible units are commutative.

Consider the operations instead as a list, such as e.g. "add(x); sub(y)" (in reality, of course, we'd record each transaction into a ledger, and store a lot more information about each).

See that list as an expression with ";" as the operator. This is commutative - the operands to ";" can be reordered at will. That ability to reorder individual operations is what we're talking about in this context, not the details of each operation.

Re: Consistency is Consistently Undervalued

#127

Post by vidarh »

superuser2 wrote: post #103
vidarh wrote: post #25

My opinion is exactly opposite: Consistently is overvalued. Requiring consistency in distributed system generally leads to designs that reduces availability. Which is one of the reasons that bank transactions generally do not rely on transactional updates against your bank. "Low level" operations as part of settlement may us transactions, but the bank system is "designed" (more like it has grown by accretion) to func…

>designs that reduces availability. It's a slimy abuse of language to call a service "available" when it's giving incorrect output and corrupting its persistent state. I can write a program with lots of nines that is blazingly faster than its competitors if its behavior is allowed to be incorrect.

A lot of the time there is no "correct" output.

E.g. a search result from a search engine is outdated the moment it has been generated - new pages come into existence all the time, and besides the index is unlikely to be up todate.

What matters is whether "close enough" is sufficient, and a large proportion of time it is: E.g. for the search engine example don't care if I get every single page that matched my query right at the moment I pressed submit. I care that I find something that meets my expectations.

Getting "good enough" results and getting them at a reasonable time beats not getting results or waiting ages because the system is insisting on maintaining global consistency.

Yes, there are cases where "good enough" means you need global consistency, but they are far fewer than people tend to think.

Re: Consistency is Consistently Undervalued

#128

Post by vidarh »

wpietri wrote: post #88
vidarh wrote: post #58

Earlier quoted context omitted.

> Do most distributed systems have the ability to detect inconsistency? They do if you build them to have them. Your bank account works without requiring consistency because every operation is commutative: You never do anything that says "set account to value x". You have operations like "deposit x" and "withdraw x". For operations that require you move money from one account to another, you need to guarantee that bo…

I don't have much to add to your excellent post; I just wanted to underline this: > Log events durably first, and then act on the events. Nothing has changed my view of how to build systems more than the log-then-act paradigm. So many problems go away when I start from that perspective. I think it took me so long to see it because so many common tools work against it. But those tools were built in an era when most sy…

Exactly, not least because it is fantastic for debugging to be able to roll a copy of a sub-system back and re-process operations and watch exactly what breaks instead of trying to figure things out from a broken after-the-fact system state.

Re: Consistency is Consistently Undervalued

#129

Post by vidarh »

SomeStupidPoint wrote: post #74
knucklesandwich wrote: post #66

Earlier quoted context omitted.

Apologies, I didn't mean it as a critique of your post. What I meant was that its often used as a strawman example of why you don't need consistency[1]. Bank transactions are extremely simple compared to the majority of things we build on top of databases. [1]: http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on...

Also, banks have very low consistency requirements. Almost every transaction is of the form: 1. estimate authorization; 2. present request for settlement; 3. settle during batch processing; 4. any additional resolution steps (eg, returned checks) The estimator used in 1 need not be fully consistent because a) as long as the periods of inconsistency are low (eg, The actual settlement is run during a special batch tran…

> The estimator used in 1 need not be fully consistent because a) as long as the periods of inconsistency are low (eg, A lot of transactions that need to be reconciled happens without involving 1. e.g. checks and offline card transactions. Even ATMs can often continue to dispense cash when their network is offline, though there arguable 1 applies in the form of maximum withdrawal limits.

But in general I agree with you - their system is built around having accepted, calculated and accounted for the risks rather than try to mitigate every possible risk. Which makes sense because a large proportion of a banks business is to understand and manage risk, and some risks are just happens cheaper to accept than to mitigate via technology or business processes (and the ones that aren't, or aren't any more, are where you'll see banks push new technology, such as making more settlements happens faster and fully electronically)

Re: Consistency is Consistently Undervalued

#130

Post by vidarh »

4ad wrote: post #79
vidarh wrote: post #25

My opinion is exactly opposite: Consistently is overvalued. Requiring consistency in distributed system generally leads to designs that reduces availability. Which is one of the reasons that bank transactions generally do not rely on transactional updates against your bank. "Low level" operations as part of settlement may us transactions, but the bank system is "designed" (more like it has grown by accretion) to func…

The bank example is terrible. The inconsistency is an absolute PITA for the consumer. The delays are are a terrible, terrible user experience, and the consumer has to spend more money in the end. It's extremely expensive, both in money the bank has to spend on auditing and in the legal framework that has to be put in place. The bank example is the perfect argument for consistency, not against.

It's a perfect argument for shorter transaction processing times. The bank inconsistency is there because historically it was impossible to guarantee consistency without making the processing times far worse. E.g. want to make a payment via check? Sorry, need to send a courier to the branch holding your account to verify the funds are there, and then afterwards send a courier to inform of the updated account state before another payment would be allowed.

But the shorter you make the processing time, the less benefit you get from introducing consistency requirements, to the point where the thing that is important to address is the remaining cases where processing transactions take time, not the consistency (at least as long as banks also don't take the piss by taking advantage of the lax consistency to charge unwarranted overdraft fees).

Post reply on HN