Consistency is Consistently Undervalued • Hacker News Board

Re: Consistency is Consistently Undervalued

#91

Post by nickpsecurity » Sat, Sep 17, 2016, 5:36 PM UTC

marknadal wrote: post #86

Disclaimer: I work on distributed systems and have spoken around the world on them, so I am very biased. I think something a lot of people miss is that the universe itself is not strongly consistent. This is Einstein's theory of relativity. It fundamentally takes time for information to travel. So if strong consistency is not viable, even at a physics level, what can we do instead? That is why our team here at http:/…

"This is Einstein's theory of relativity. It fundamentally takes time for information to travel."

Which is why Google's hack in Spanner and F1 RDBMS of simply using or waiting out the known time delay was so clever. Also relies on a tool that was a test of Einstein's theory. ;)

"That is why our team here at http://gun.js.org/ believe in CRDTs."

Thanks for the link. I'll look into it. Plus the liberal license that gives it a chance of making a dent in commercial sector or in combo with BSD/Apache projects.

Re: Consistency is Consistently Undervalued

#92

Post by wpietri » Sat, Sep 17, 2016, 5:42 PM UTC

knucklesandwich wrote: post #82

vidarh wrote: post #68

Earlier quoted context omitted.

> You can't do this in general, because we often want to build applications where the user expects their actions won't be reordered. The key being they don't expect their actions to be reordered. In most systems this is trivially handled by logging event time as seen from the users system and ensure the user sees a consistent view of their own operations . There are exceptions, naturally, but the point is not that yo…

> Your privacy settings is a local concern to your account, it most certainly does not need to be globally consistent. All you need to do to avoid problems with that is to ensure consistency in routing requests for a given session, and enforce local consistency of ordering within a given user account or session depending on requirements. I think you're misunderstanding what the problem is here. I want global consiste…

Do you really need global consistency for that? I think you can get by with user-consistent ordering as long as your data model's right.

It sounds like you're saying that the friend-state for a given post are part of the permissions for the post, and therefore part of the post itself. In which case, when a user posts, I'd include the version number of their friend-state in the post's permissions. Then on view attempts, I make sure that the friend-state I'm working with is at least as new as the friend-state referred to by the post.

Re: Consistency is Consistently Undervalued

#93

Post by chajath » Sat, Sep 17, 2016, 5:45 PM UTC

To actually do a distributed transaction, I would look into algorithms such as 2PC or 3PC. Although before going there, I would seriously consider consolidating different backends into one scalable option (banking transaction example is a bit contrived, although I gather the author is just trying to make a point).

At the service level integration, we can leverage a reliable message queue middleware to make sure a task is eventually delivered and handled (or be put in a dead letter queue so we can do a batch clean up)

Also as a general principle, I would make each of those sub-transactions to be idempotent, so that retrying multiple times won't hurt, and there would be a natural way of picking the winner if there are conflicting ongoing commit / retry attempts.

Re: Consistency is Consistently Undervalued

#94

Post by uryga » Sat, Sep 17, 2016, 6:00 PM UTC

vidarh wrote: post #58

kahirsch wrote: post #48

Earlier quoted context omitted.

As you say, the low-level operations in banking do provide atomicity, which are often lacking in distributed databases. Also, the banking system does have the system of compensating transactions (e.g. returned checks) when constraints are violated, as well as an extensive system of auditing to maintain consistency. Do most distributed systems have the ability to detect inconsistency? How do they manage to correct pro…

> Do most distributed systems have the ability to detect inconsistency? They do if you build them to have them. Your bank account works without requiring consistency because every operation is commutative: You never do anything that says "set account to value x". You have operations like "deposit x" and "withdraw x". For operations that require you move money from one account to another, you need to guarantee that bo…

Correct me if I'm wrong, but I feel like bank transactions aren't always commutative. Consider:

balance = 0 balance += 100 balance -= 50

You can't swap the order of addition and subtraction, because that would mean withdrawing from an empty account, and afaik you can't do that. Is there something I'm missing?

Re: Consistency is Consistently Undervalued

#95

Post by knucklesandwich » Sat, Sep 17, 2016, 6:02 PM UTC

wpietri wrote: post #92

knucklesandwich wrote: post #82

Earlier quoted context omitted.

> Your privacy settings is a local concern to your account, it most certainly does not need to be globally consistent. All you need to do to avoid problems with that is to ensure consistency in routing requests for a given session, and enforce local consistency of ordering within a given user account or session depending on requirements. I think you're misunderstanding what the problem is here. I want global consiste…

Do you really need global consistency for that? I think you can get by with user-consistent ordering as long as your data model's right. It sounds like you're saying that the friend-state for a given post are part of the permissions for the post, and therefore part of the post itself. In which case, when a user posts, I'd include the version number of their friend-state in the post's permissions. Then on view attempt…

There are definitely solutions for handling this with consistency semantics that are something weaker than linearizable consistency. But they usually become expensive in other ways and loose a lot of the availability benefits to recover the consistency properties you do need. Most of the models we're even talking about here are something in the realm of sequential or causal consistency, which are still considered strong consistency models (and are usually implemented on top of linearizable algorithms that allow for weakening invariants that protect against witnessing inconsistency. spanner is an example of this).

Your friend-state solution only protects the visibility of the post if the friend-state between the two users has not changed more recently than both the post and your version.

I.E.

Friend state 0 is we're friends.

Friend state 1 is we're not friends.

Friend state 2 is we're friends.

Friend state 3 is we're not friends.

I have friend state 2, and the post is at state 1. I accept my version incorrectly and am able to view the post, despite the fact that we're no longer friends.

Re: Consistency is Consistently Undervalued

#96

Post by dang » Sat, Sep 17, 2016, 6:09 PM UTC

yyyuuu wrote: post #3

Continuous as the stars that shine And twinkle on the milky way, They stretched in never-ending line Along the margin of a bay: Ten thousand saw I at a glance, Tossing their heads in sprightly dance.

We're always up for the Romantics (though I prefer Coleridge), but maybe not like this. Your comments have been inappropriate for Hacker News lately, especially the one where you impersonated another user. This sort of pattern will get an account banned, so please stick to civil, substantive comments only.

Re: Consistency is Consistently Undervalued

#97

Post by dasil003 » Sat, Sep 17, 2016, 6:11 PM UTC

vidarh wrote: post #25

My opinion is exactly opposite: Consistently is overvalued. Requiring consistency in distributed system generally leads to designs that reduces availability. Which is one of the reasons that bank transactions generally do not rely on transactional updates against your bank. "Low level" operations as part of settlement may us transactions, but the bank system is "designed" (more like it has grown by accretion) to func…

I guess whether it's overvalued or undervalued depends on whom we're talking about. Your point is well taken, but I think the real problem is that inexperienced programmers want the best of both worlds. In other words, they design systems that require consistency to function correctly because it's easier, but then they do not actually guarantee said consistency, because hey, schemaless is so much faster than boring old SQL.

The way I like to think about it (and mind you, I'm an early-stage kind of guy, so I rarely work on huge systems) is that consistency guarantees give you a lot of leverage, and while it's almost unattainable in the real world, in the computing world we have extremely powerful CPUs and networks that allow communication between any two points on the globe in less than a second. There is frankly a huge number of applications that can be built on an ACID database and never have to worry about the offline or distributed use cases that would require building a proper distributed system.

Re: Consistency is Consistently Undervalued

#98

Post by knucklesandwich » Sat, Sep 17, 2016, 6:31 PM UTC

SomeStupidPoint wrote: post #67

knucklesandwich wrote: post #54

Earlier quoted context omitted.

The bank example really needs to die, its not reflective of what people often use databases for. 1. Bank transactions are easy to express as a join-semilattice[1], which means its intrinsically easy to build a convergent system of bank accounts in a distributed environment. Many problems are not easily expressible as these and usually trade off an unacceptable amount of space to formulate as a join-semilattice and st…

> There is no perception of reordering my intentions in the real world. Never tried to type something by thinking only of the word (rather than specifically the sequence of key strokes) only to have your fingers execute the presses in slightly the wrong order -- giving you something like "teh" instead of "the"? I have, a lot actually, and if I'm in a hurry or distracted, it can even pass by unnoticed, leaving me with…

It's a cool metaphor, and I'm sure there are occasions where its easy to pass this off as innocuous. But inconsistent semantics can produce blemishes and bugs that can be difficult to anticipate and can very negatively impact user experience if you aren't careful. The typo explanation is acceptable when I just see some comment I posted being reordered after a page refresh. It's less so if I get locked out of my account because my password change hasn't fully replicated to all servers (and even less so if my consistency model doesn't converge).

To be clear: there's an economic tradeoff to make here that isn't identical at all scales. At a certain point, lost revenue due to user dropoff from availability is more expensive than users who stop using a site due to a frustrating experience (and sometimes that scale itself is the compelling feature able to stave off a user from giving up on your app completely). But in most cases, I think its advisable to start with stronger consistency semantics and only weaken them when a need presents itself. Hell, Spanner is the foundation for Google's ad business, if that's not living proof that you can preserve strong consistency semantics and keep your system sufficiently available, I'm not sure what is.

Re: Consistency is Consistently Undervalued

#99

Post by knucklesandwich » Sat, Sep 17, 2016, 7:03 PM UTC

uryga wrote: post #94

vidarh wrote: post #58

Earlier quoted context omitted.

> Do most distributed systems have the ability to detect inconsistency? They do if you build them to have them. Your bank account works without requiring consistency because every operation is commutative: You never do anything that says "set account to value x". You have operations like "deposit x" and "withdraw x". For operations that require you move money from one account to another, you need to guarantee that bo…

Correct me if I'm wrong, but I feel like bank transactions aren't always commutative. Consider: balance = 0 balance += 100 balance -= 50 You can't swap the order of addition and subtraction, because that would mean withdrawing from an empty account, and afaik you can't do that. Is there something I'm missing?

There are a couple ways you could do this, though according to jerf in a comment below, banks often don't implement transactions as commutative.

Most commonly things like this are done by somehow reifying the operation as data and attaching some information to determine ordering. One way would be to attach a timestamp to bank transactions. By itself, that would be enough to ensure bank transactions commute (but without bounds on clock synchronization, wouldn't guarantee that you see those operations in the order you issue them). You could also require a single consistent read to an account to get a state version number and attach an incremented version of that in your transaction. Ties in both cases could be resolved by account id of the person initiating the transaction.

Re: Consistency is Consistently Undervalued

#100

Post by SomeStupidPoint » Sat, Sep 17, 2016, 7:07 PM UTC

knucklesandwich wrote: post #98

SomeStupidPoint wrote: post #67

Earlier quoted context omitted.

> There is no perception of reordering my intentions in the real world. Never tried to type something by thinking only of the word (rather than specifically the sequence of key strokes) only to have your fingers execute the presses in slightly the wrong order -- giving you something like "teh" instead of "the"? I have, a lot actually, and if I'm in a hurry or distracted, it can even pass by unnoticed, leaving me with…

It's a cool metaphor, and I'm sure there are occasions where its easy to pass this off as innocuous. But inconsistent semantics can produce blemishes and bugs that can be difficult to anticipate and can very negatively impact user experience if you aren't careful. The typo explanation is acceptable when I just see some comment I posted being reordered after a page refresh. It's less so if I get locked out of my accou…

> The typo explanation is acceptable when I just see some comment I posted being reordered after a page refresh. It's less so if I get locked out of my account because my password change hasn't fully replicated to all servers (and even less so if my consistency model doesn't converge).

The typo model is merely meant to explain why it happened. You'd be no more happy if you were locked out of a door because a locksmith fat-fingered the code on setting a pin. Not all typos are harmless, and that wasn't the point of my analogy.

Helping people understand why these technology mistakes happen -- through intuitive models like typos -- helps us discuss when mistakes are and aren't acceptable, and how we're going to address the cases where they're not. It simply gives me a way to discuss it with non-technical people: under what situations would a person doing the computer's job making a typo be catastrophic instead of merely annoying?

Most managers or clients can understand that question, at an intuitive level.

> But in most cases, I think its advisable to start with stronger consistency semantics and only weaken them when a need presents itself.

Completely agree! Unless you have a specific reason it's necessary for you, you likely don't have to actually address the inconsistency directly and are better off letting the database handle that for you.