Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data consistency is overrated

two-wrongs.com

Post reply on HN
Print view View original

61–70 of 72 posts

Re: Data consistency is overrated

#61

Post by dataflow »

kqr wrote: post #60
dataflow wrote: post #55

Earlier quoted context omitted.

That's an incredibly misleading way to phrase it. The counting algorithm is exact. Errors creep in during execution, resulting in an approximate result . An approximation algorithm is a whole different beast, and that's the point of the discussion here.

Does really how the error comes into life matter more than the magnitude of the error?

It's pretty easy to find out - go pitch some approximate vote counting algorithm (say, 50% random subsampling to get 2x speed? or whatever) to random people on the street, and ask them if they would care if the government decided the president that way.

Re: Data consistency is overrated

#62

Post by esailija »

StreamBright wrote: post #9
fbdab103 wrote: post #8

Earlier quoted context omitted.

Airlines are notorious for over-booking available seats and dealing with the fallout.

Which is done purposefully and not by data in-consistency at all.

No, it's because of CAP forces to make a compromise. If there was a way to be available and consistent there would not be overbooking. But they chose availability because losing sales is worse than dealing with overbooking.

Exactly same with Amazon, they will not bother checking that the product is actually available because then you will miss the sale when the warehouse service is down or slow, which it would be all the time because it has to be a single place of failure. Its better to refund orders for non existent items than to miss sales.

Consistency requires single source of truth which implies single place of failure. There is tremendous cost to it, it's not done for no reason like you say.

Re: Data consistency is overrated

#63

Post by josephg »

zvrba wrote: post #58
josephg wrote: post #28

Earlier quoted context omitted.

Hm, I'd cut the cake a little differently. I think there's two kinds of consistency we can strive for: - Strict consistency. Every pointer in my b-tree must point to a b-tree node, and not random data which could cause the program to crash. In a financial world, a bank should never print money. - Fuzzy "good enough" consistency. In the examples in the article, all of the financial transactions should end up close eno…

> Null pointers take something that should be strictly consistent (references) and make it fuzzy. I'd disagree; this is still strict consistency, at least in managed languages like Java or C#. (Coming from someone who has fully embraced nulls instead of fighting them.) "Strict consistency" breaks in C and C++ where a pointer can either point to 1) a valid object, 2) be null, 3) point to an invalid address (i.e., dere…

>> Null pointers take something that should be strictly consistent (references) and make it fuzzy.

> I'd disagree; this is still strict consistency, at least in managed languages like Java or C#

The semantics are well defined, but the program that lives in my head (where the pointer is never null) can be subtly different from the program that runs on my computer. On the computer, the pointer can and will very occasionally be null, because I have a bug somewhere else in my code that the compiler isn't smart enough to point out to me.

That window of fuzzyness is wide enough to fit a peak hour traffic jam full of bugs.

> (Coming from someone who has fully embraced nulls instead of fighting them.)

Everyone who programs in C, Java, Javascript, C# and all the other languages with null pointers feel like they "embrace nulls rather than fight them". And yet, memory bugs still make up ~60% of the critical security vulnerabilities in Chrome and other systems software. I've seen so many crashes in application software over the years from null pointer exceptions (or the C/C++ equivalents).

I've been programming for 30 years and I still make occasional mistakes in these languages ending up with null pointer related errors.

I think its fair to say that moving away from implicitly nullable types lowers our collective defect rate. Language authors are on board too. Haskell, Swift, Rust, Typescript and even (I think) C++ now have non-nullable types in their type system.

Even if you feel like you're too smart to need the guard rails of non-nullable references, I'm definitely not that smart. And chances are, you run my code. Tightening up this stuff lowers the chance that my software crashes while you're using it.

Re: Data consistency is overrated

#64

Post by zvrba »

josephg wrote: post #63
zvrba wrote: post #58

Earlier quoted context omitted.

> Null pointers take something that should be strictly consistent (references) and make it fuzzy. I'd disagree; this is still strict consistency, at least in managed languages like Java or C#. (Coming from someone who has fully embraced nulls instead of fighting them.) "Strict consistency" breaks in C and C++ where a pointer can either point to 1) a valid object, 2) be null, 3) point to an invalid address (i.e., dere…

>> Null pointers take something that should be strictly consistent (references) and make it fuzzy. > I'd disagree; this is still strict consistency, at least in managed languages like Java or C# The semantics are well defined, but the program that lives in my head (where the pointer is never null) can be subtly different from the program that runs on my computer. On the computer, the pointer can and will very occasio…

So what to do about stuff such as lazy creation of objects?

- Option 1: nullable reference and if the program crashes, you have a bug. (That should be caught during testing.)

- Option 2: Maybe, making the rest of the code unreadable. Because

- Option 2a: "Force-extract" the value and get another exception if the value is not initialized

- Option 2b: Do "proper" pattern-match and do ??? if the value is not initialized.

So I strongly prefer a plain nullable reference. If I dereference it, it raises an exception that gets handled at top-level _just like any other exception due to any other kind of bug_. There's no reason that, in a properly structured program, dereferencing null should crash the whole program. (In a managed language. Unmanaged languages have their own problems - e.g., impossible to determine if a non-null pointer is at all valid. Though OS _does_ let you handle segfaults, if you want to go that way. Windows structured exceptions are much more flexible there than anything available on unix.)

Re: Data consistency is overrated

#65

Post by LAC-Tech »

deathanatos wrote: post #57
LAC-Tech wrote: post #30

Wisdom from 40 years ago: Network partitioning can completely destroy mutual consistency in the worst case, and this fact has led to a certain amount of restrictiveness, vagueness, and even nervousness in past discussions, of how it may be handled. In some environments it is desirable or necessary to permit users to continue modifying resources such as files when the network is partitioned. A network operating system…

> wait for consensus on all nodes... You can (e.g., using something like Raft) permit writes on the side of the netsplit that can establish quorum (if it exists), which is much less than all nodes.

Good point.

Re: Data consistency is overrated

#66

Post by StreamBright »

esailija wrote: post #62
StreamBright wrote: post #9

Earlier quoted context omitted.

Which is done purposefully and not by data in-consistency at all.

No, it's because of CAP forces to make a compromise. If there was a way to be available and consistent there would not be overbooking. But they chose availability because losing sales is worse than dealing with overbooking. Exactly same with Amazon, they will not bother checking that the product is actually available because then you will miss the sale when the warehouse service is down or slow, which it would be all…

> there would not be overbooking.

This is not the case at all.

Airlines operate on the principle that a certain percentage of customers are no show. They would like to fly the planes fully booked so they allow some overbooking and they are controlling how much it gets overbooked.

If it was up to CAP there would be 200% overbooking for certain flights and the airline would go bust within a year.

Re: Data consistency is overrated

#67

Post by naijaboiler »

dataflow wrote: post #50
gorbachev wrote: post #5

I once worked on a platform producing analytics using data that, at its source, was manually typed in by people. My product managers would insist we do distinct counts on the aggregates instead of using probabilistic algorithms, because we "needed" the absolute 100% accurate output. No matter how many times I would explain the data was never 100% accurate to begin with and that the error rate using HyperLogLog wouldn…

I could see myself being like your PM in some situations, depending on the nature of the data and how it was being used. To give an analogy that might perhaps help with understanding the sentiment, imagine how your fellow citizens would feel if the government used some {hot approximation algorithm} for counting votes in the next election, rather than a straightforward count.

vote counts are approximate. we just make it seem like they are exact!

Re: Data consistency is overrated

#68

Post by bcherny »

riku_iki wrote: post #51
bcherny wrote: post #47

Earlier quoted context omitted.

Why would your PM dictate the technical implementation? This seems like something better left to engineers.

distinct vs approximate user facing result seems like product question.

The commenter seemed to indicate that even the exact result wasn’t very accurate. Perhaps a failure on their part to communicate this nuance well, or understand the business context.

Re: Data consistency is overrated

#69

Post by ddulaney »

GauntletWizard wrote: post #37
bcrosby95 wrote: post #29

Earlier quoted context omitted.

Deletion is acceptable, and if you have everything in a consistent system they will both exist or not. "Nonconsistency advocacy" doesn't make a lot of sense to me here. Are you advocating not relying on consistency in systems that guarantee consistency? That is a waste of time. Are you advocating eschewing consistent systems? Well, then you have more work, so only if I need to. And yes, you should handle data errors…

My point is: Removing a music track from the list of playable tracks is not a good reason to remove it from playlists. Having dangling pointers there, and handling them appropriately is the right thing to do. This is true in a ton of cases where neither soft-delete or cascading delete make sense. Rarely, if ever, have I actually encountered the latter.

Hmm... That's an interesting example. It definitely makes sense to enable features like that.

However, I'm not sure it's a place where you want to abandon foreign-key consistency. To display the song on the playlist (greyed-out, Spotify-style) you need its name and info, and you probably want to keep artist and album links valid as well. It sounds like this is actually a perfect use-case for a soft-delete: keep the song metadata in the songs table and mark it as nonplayable. This keeps the artist page, album page, and any links to the song itself valid, just greyed out when you get there.

Re: Data consistency is overrated

#70

Post by jongjong »

yxhuvud wrote: post #59
jongjong wrote: post #45

I find the example given by the author about financial inconsistency to be alarming. When you consider the fact that the monetary system is made up of large numbers of participants and all of them are constantly making small mistakes... Surely it can accumulate into significant discrepancies over time. Imagine that you create a multiplayer MMORPG video game with a 'limited' number of virtual gold coins and you set th…

A mmorpg would have user churn though, so even if it aquired more users to offset that the end result would be less coins in circulation. In other words: The system is already broken by design. Using a single consistent server would not change that. I also wonder why you think money creation isn't supposed to be in private hands. It needs to be regulated with enforcement, but at the end of the day it is the only reas…

I think it would make an even bigger difference with churn. The currency which is managed by a single system has a much higher chance of being deflationary in practice (due to churn taking coins out of circulation). The currency managed by 1000 different systems is very unlikely to be deflationary; chances are, at least one of those 1000 systems is going to go rogue (e.g because of a bug or hack) and start issuing coins which will propagate to all other systems which trust it. It may be difficult to figure out which one started breaking the rules and once they found out, there would be no incentive to report it or fix the problem.
Post reply on HN