Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data consistency is overrated

two-wrongs.com

Post reply on HN
Print view View original

51–60 of 72 posts

Re: Data consistency is overrated

#51

Post by riku_iki »

bcherny wrote: post #47
gorbachev wrote: post #5

I once worked on a platform producing analytics using data that, at its source, was manually typed in by people. My product managers would insist we do distinct counts on the aggregates instead of using probabilistic algorithms, because we "needed" the absolute 100% accurate output. No matter how many times I would explain the data was never 100% accurate to begin with and that the error rate using HyperLogLog wouldn…

Why would your PM dictate the technical implementation? This seems like something better left to engineers.

distinct vs approximate user facing result seems like product question.

Re: Data consistency is overrated

#52

Post by riknos314 »

The important thing to consider is: What tradeoffs are required to achieve consistency for your use case?

Delivering consistency in either the CAP or ACID sense requires some amount of constraints on other aspects of the system. Sometimes making this tradeoff it worth it, sometimes it isn't.

The key is to make the value analysis for each problem you face so you can determine the best overall solution to the problem at hand.

Re: Data consistency is overrated

#53

Post by kqr »

dataflow wrote: post #50
gorbachev wrote: post #5

I once worked on a platform producing analytics using data that, at its source, was manually typed in by people. My product managers would insist we do distinct counts on the aggregates instead of using probabilistic algorithms, because we "needed" the absolute 100% accurate output. No matter how many times I would explain the data was never 100% accurate to begin with and that the error rate using HyperLogLog wouldn…

I could see myself being like your PM in some situations, depending on the nature of the data and how it was being used. To give an analogy that might perhaps help with understanding the sentiment, imagine how your fellow citizens would feel if the government used some {hot approximation algorithm} for counting votes in the next election, rather than a straightforward count.

Vote counts are already approximate. That's why close results are re-counted using slower methods.

(Additionally, I would argue more generally that vote counts should be approximate. If a vote is close to tied then we run the largest risk of making an incorrect decision. Better to continue debate and/or refine the proposal until it has wider acceptance. This does not work for selecting government with term limits obviously.)

Re: Data consistency is overrated

#54

Post by kqr »

layer8 wrote: post #24
legulere wrote: post #3

Data consistency comes at a cost, but eschewing it does as well. Most of the time the performance impact of data consistency does not matter, potentially introducing heisenbugs in your system can come at a huge cost though.

And also, eventual consistency means that you need to make sure that data eventually does become consistent, and not just give up on consistency.

How do you make sure of that? With atomic consistency guarantees you can "just" roll back the transaction -- nobody else would have seen any values contained in it. In an eventually consistent system some parts will have started building on the new values so you can't roll them back. You have to give up on consistency and achieve it only in a "best effort" sense.

Re: Data consistency is overrated

#55

Post by dataflow »

kqr wrote: post #53
dataflow wrote: post #50

Earlier quoted context omitted.

I could see myself being like your PM in some situations, depending on the nature of the data and how it was being used. To give an analogy that might perhaps help with understanding the sentiment, imagine how your fellow citizens would feel if the government used some {hot approximation algorithm} for counting votes in the next election, rather than a straightforward count.

Vote counts are already approximate. That's why close results are re-counted using slower methods. (Additionally, I would argue more generally that vote counts should be approximate. If a vote is close to tied then we run the largest risk of making an incorrect decision. Better to continue debate and/or refine the proposal until it has wider acceptance. This does not work for selecting government with term limits obv…

That's an incredibly misleading way to phrase it. The counting algorithm is exact. Errors creep in during execution, resulting in an approximate result. An approximation algorithm is a whole different beast, and that's the point of the discussion here.

Re: Data consistency is overrated

#56

Post by kqr »

jongjong wrote: post #45

I find the example given by the author about financial inconsistency to be alarming. When you consider the fact that the monetary system is made up of large numbers of participants and all of them are constantly making small mistakes... Surely it can accumulate into significant discrepancies over time. Imagine that you create a multiplayer MMORPG video game with a 'limited' number of virtual gold coins and you set th…

The power of money creation is in the hands of private institutions more explicitly than that. When a bank lends you money, they are just incrementing the money counter in your bank account. They cannot print bills, but they do for all practical intents create money.

https://www.bankofengland.co.uk/explainers/how-is-money-crea...

Re: Data consistency is overrated

#57

Post by deathanatos »

LAC-Tech wrote: post #30

Wisdom from 40 years ago: Network partitioning can completely destroy mutual consistency in the worst case, and this fact has led to a certain amount of restrictiveness, vagueness, and even nervousness in past discussions, of how it may be handled. In some environments it is desirable or necessary to permit users to continue modifying resources such as files when the network is partitioned. A network operating system…

> wait for consensus on all nodes...

You can (e.g., using something like Raft) permit writes on the side of the netsplit that can establish quorum (if it exists), which is much less than all nodes.

Re: Data consistency is overrated

#58

Post by zvrba »

josephg wrote: post #28
ddulaney wrote: post #7

I think there are two different kinds of consistency, and it's important to not conflate them. There's consistency that's internal to a system. Do all of the foreign keys line up correctly? Have I lost any data that was provided to me? Here, we can aspire to be 100% correct. I don't think the examples in this article conflict with that. Then there's consistency that's external to a system. This can be between this sy…

Hm, I'd cut the cake a little differently. I think there's two kinds of consistency we can strive for: - Strict consistency. Every pointer in my b-tree must point to a b-tree node, and not random data which could cause the program to crash. In a financial world, a bank should never print money. - Fuzzy "good enough" consistency. In the examples in the article, all of the financial transactions should end up close eno…

> Null pointers take something that should be strictly consistent (references) and make it fuzzy.

I'd disagree; this is still strict consistency, at least in managed languages like Java or C#. (Coming from someone who has fully embraced nulls instead of fighting them.)

"Strict consistency" breaks in C and C++ where a pointer can either point to 1) a valid object, 2) be null, 3) point to an invalid address (i.e., dereferencing it crashes the program). Cases 1 and 2 can simply be distinguished by simple != 0 check. There is no (direct, simple, portable, defined by the language) check to distinguish between cases 1 and 3, which is where the inconsistency arises.

Re: Data consistency is overrated

#59

Post by yxhuvud »

jongjong wrote: post #45

I find the example given by the author about financial inconsistency to be alarming. When you consider the fact that the monetary system is made up of large numbers of participants and all of them are constantly making small mistakes... Surely it can accumulate into significant discrepancies over time. Imagine that you create a multiplayer MMORPG video game with a 'limited' number of virtual gold coins and you set th…

A mmorpg would have user churn though, so even if it aquired more users to offset that the end result would be less coins in circulation. In other words: The system is already broken by design. Using a single consistent server would not change that.

I also wonder why you think money creation isn't supposed to be in private hands. It needs to be regulated with enforcement, but at the end of the day it is the only reasonably efficient way to couple the productive ability of society with its consumption. The world revolves around promises. Not all will be kept, but the system as a whole is built to handle that and eventually resolve inconsistencies.

That is also why autocratic and corrupt systems are so bad, they put the self correcting parts out of order.

Re: Data consistency is overrated

#60

Post by kqr »

dataflow wrote: post #55
kqr wrote: post #53

Earlier quoted context omitted.

Vote counts are already approximate. That's why close results are re-counted using slower methods. (Additionally, I would argue more generally that vote counts should be approximate. If a vote is close to tied then we run the largest risk of making an incorrect decision. Better to continue debate and/or refine the proposal until it has wider acceptance. This does not work for selecting government with term limits obv…

That's an incredibly misleading way to phrase it. The counting algorithm is exact. Errors creep in during execution, resulting in an approximate result . An approximation algorithm is a whole different beast, and that's the point of the discussion here.

Does really how the error comes into life matter more than the magnitude of the error?
Post reply on HN