Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There Is No Now – Problems with Simultaneity in Distributed Systems

queue.acm.org

Post reply on HN
Print view View original

31–40 of 72 posts

Re: There Is No Now – Problems with Simultaneity in Distributed Systems

#31

Post by marknadal »

xxxyy wrote: post #28
logicallee wrote: post #19

Earlier quoted context omitted.

I'm not an expert, and it sounds like you are, so I appreciate your feedback here: what do you even mean by a consistent state? even in theory a person initiating a new additional record in Auckland, New Zealand at the same time somebody iniatiates a change in Gibraltar or London (which are antipodal to the former[1]) 66 milliseconds away, cannot have a confirmation in less than 120 milliseconds, right? So do you jus…

Given a set of perfectly synchronized distributed clocks you may not even have to wait the 66ms. Incoming transactions (both local and remote) go into the write-ahead log, their ordering is given by the timestamps (which are consistent, because clocks are synchronized). Periodic heartbeats from other nodes give you a green light to commit or abort parts of the write-ahead log into permanent database. All nodes will m…

Warning, this is a very dangerous practice. The whole point of the article states that machines do fail and that networks are unreliable. He had a specific section in their that even Spanner has a 7ms uncertainty on time.

Therefore: You CANNOT trust timestamps or your clock.

Re: There Is No Now – Problems with Simultaneity in Distributed Systems

#33

Post by tlarkworthy »

nostrademons wrote: post #22
dragonwriter wrote: post #14

Earlier quoted context omitted.

> we have very strict laws of physics supporting the concept of now (sans relativity) Laws of physics "sans relativity" aren't the actual laws of physics in our universe. There very much is no now except the now that is also here . Its quite accurate to say that simultaneity does not exist in distributed systems, and simultaneity is less valid as even an approximation the more widely distributed a system is.

To add some context & numbers to dragonwriter's point, the speed-of-light delay from New York to San Francisco is about 21ms [1]. This is about 5 disk seeks, 1200 random SSD reads, 200K main memory reads (without caching), or 10M CPU cycles [2]. Speed of light delays absolutely matter in a distributed system. [1] http://chimera.labs.oreilly.com/books/1230000000545/ch01.htm... [2] http://www.eecs.berkeley.edu/~rcs/res…

that's transmission delays not relativistic effects. The idea of spanner is you have a timestamp of when it was decided to commit. The quorum knows they can't contact each other quickly but they trust each others timestamps and resolve conflicts based on the trusted commit times (which are accurate through hardware). Transmission delays don't undermine the fact there is a very real concept of ordered time in the physical world which is exploitable[1]. Spanner exploits it faster than transmission delays, but with a clock error of 10 ms or something. We have better clocks than that so its probably going to improve...

[1] sans relativity effects which are TINY, and not the limiting factor at the moment.

Re: There Is No Now – Problems with Simultaneity in Distributed Systems

#35

Post by logicallee »

bdamm wrote: post #24
logicallee wrote: post #19

Earlier quoted context omitted.

I'm not an expert, and it sounds like you are, so I appreciate your feedback here: what do you even mean by a consistent state? even in theory a person initiating a new additional record in Auckland, New Zealand at the same time somebody iniatiates a change in Gibraltar or London (which are antipodal to the former[1]) 66 milliseconds away, cannot have a confirmation in less than 120 milliseconds, right? So do you jus…

There are many systems which can work just fine in an eventually-consistent manner. A database of people (customers, users, etc) is a classic example of such a thing. In general I think consistency is over valued. There are plenty of cases where it is important. Lots of people are brainwashed in college to think that all data must be consistent all the time, and that's just not necessary.

>Lots of people are brainwashed in college to think that all data must be consistent all the time, and that's just not necessary.

I knew it! So a foolish consistency is the hobgoblin of little minds.

Re: There Is No Now – Problems with Simultaneity in Distributed Systems

#36

Post by marknadal »

stcredzero wrote: post #26
jchrisa wrote: post #20

Earlier quoted context omitted.

If you want to let a mature open source sync system do the heavy lifting we've recently added Unity 3D support to Couchbase Lite. http://developer.couchbase.com/mobile/unity/ Hello world is a game where players drop shapes into a world and they automatically sync / appear on other devices. Sorry about the plug -- great article Justin! I sent it to my team because I think it does a great job stepping back from the buz…

Interesting. However I'm not just looking for syncing stats and inventory. What I'm talking about is designing a new game mechanic around the realities of distributed gaming, then designing a more robust server around that. The server will be finalizing entity positions, damage, and collisions a half second behind "real-time," and the game mechanic will be designed around this in a way that preserves player autonomy…

@stcredzero I'm interested in building simulation systems for distributed systems, and trying to "gamify" them to teach people about these concepts in a tactile-playable way. Would you be interested in such work? It is less building a "entertainment" game and more an "educational" fun game. Shoot me an email: mark@gunDB.io

Re: There Is No Now – Problems with Simultaneity in Distributed Systems

#37

Post by olau »

logicallee wrote: post #35
bdamm wrote: post #24

Earlier quoted context omitted.

There are many systems which can work just fine in an eventually-consistent manner. A database of people (customers, users, etc) is a classic example of such a thing. In general I think consistency is over valued. There are plenty of cases where it is important. Lots of people are brainwashed in college to think that all data must be consistent all the time, and that's just not necessary.

>Lots of people are brainwashed in college to think that all data must be consistent all the time, and that's just not necessary. I knew it! So a foolish consistency is the hobgoblin of little minds.

While I agree strict consistency is probably overkill in many if not most situations, the problem with not having consistency is that it potentially makes the application logic much more complicated.

Take the database of customers - so if you don't have consistency, what happens in case someone changes the company address and another person simultaneously requests a delivery of something. Do you risk ending up with half of the old address and half of the new one on the parcel?

Note you can certainly have this problem in a consistent system too, e.g. if you make a UI without a save button where the address is changed one field at a time.

Concurrency is just intrinsically hard.

Re: There Is No Now – Problems with Simultaneity in Distributed Systems

#39

Post by rdtsc »

logicallee wrote: post #19
rdtsc wrote: post #11

Keeping an always consistent state in a large distributed you are fighting against the law of physics. Like it is mentioned, Google did it with their F1/Spanner SQL database. But that also mean GPS receivers with antennas on the roofs of data centers. Which is yet another thing that can fail and it, either by itself or in a cascade of other failures will lead to unspecified and possibly undesirable behavior. Recently…

I'm not an expert, and it sounds like you are, so I appreciate your feedback here: what do you even mean by a consistent state? even in theory a person initiating a new additional record in Auckland, New Zealand at the same time somebody iniatiates a change in Gibraltar or London (which are antipodal to the former[1]) 66 milliseconds away, cannot have a confirmation in less than 120 milliseconds, right? So do you jus…

> Do you literally add 120 milliseconds to each and every request?

Yap, you have to add all the delay until you get confirmation from the members of the cluster that the write was written. And you have to have linearizability, so that anyone reading after that (and one can argue what 'that' and where 'that' is) will now see the new state. If one of the members has failed you could potentially be stuck forever waiting. Now you also have to make sure how cluster membership and connectivity is defines and what are the possible state and transition during membership change, coupled with network partitions, coupled with hardware failures.

In other words you are fighting against the laws of physics. It is expensive and hard to do.

In case of the counters, one should ask is it worth it. Or is an CRDT based counter (that will eventually converge) good enough.

Even banks are eventually consistent. They choose to be available first. So you can withdraw $100 in New Zeland and then $100 in New York with a short period of time even if you only have $100 in your account. Inconsistency is handled later when you get a letter that your account is overdrawn.

Re: There Is No Now – Problems with Simultaneity in Distributed Systems

#40

Post by keppy »

> Another such area of work is logical time, manifest as vector clocks, version vectors, and other ways of abstracting over the ordering of events. This idea generally acknowledges the inability to assume synchronized clocks and builds notions of ordering for a world in which clocks are entirely unreliable.

Hardware is unreliable. Software is possibly less reliable. We have known that for a long time. The author talks on a conceptual level about logical time, but this concept isn't enough to understand the real challenges & possible solutions of keeping interactions in your system logically ordered in the dimension of time[0].

> You can think of coordination as providing a logical surrogate for "now." When used in that way, however, these protocols have a cost, resulting from something they all fundamentally have in common: constant communication. For example, if you coordinate an ordering for all of the things that happen in your distributed system, then at best you are able to provide a response latency no less than the round-trip time (two sequential message deliveries) inside that system.

Consensus protocols don't provide a logical surrogate for 'now', a log does that. The silver bullet for assuring that your transactions are ordered correctly is immutability[1]--"If two identical, deterministic processes begin in the same state and get the same inputs in the same order, they will produce the same output and end in the same state.[0]" It's important, from the perspective of the implementor, to understand that there are multiple pieces to this puzzle, and that each protocol has very specific details that can make or break the reliability and performance of a distributed system. This is similar to how a small bug in your cryptography code can expose the entire system to threat. Paxos itself can be implemented in a myriad of ways, and each decision the implementor makes must be well researched.

[0]http://engineering.linkedin.com/distributed-systems/log-what...

[1]http://basho.com/clocks-are-bad-or-welcome-to-distributed-sy...

Post reply on HN