FTX's Excel spreadsheet has entered the chat.
Data consistency is overrated
31–40 of 72 posts
Re: Data consistency is overrated
#32Wouldn't it be funny if the post changed every few hours?
Does the post need multiple redundant copies with high write availability?
Because a lot of data does.
Re: Data consistency is overrated
#33I think there are two different kinds of consistency, and it's important to not conflate them. There's consistency that's internal to a system. Do all of the foreign keys line up correctly? Have I lost any data that was provided to me? Here, we can aspire to be 100% correct. I don't think the examples in this article conflict with that. Then there's consistency that's external to a system. This can be between this sy…
Prof. Daniel Abadi talks about the two meanings of consistency in this blogpost: https://dbmsmusings.blogspot.com/2019/07/overview-of-consist...
I'd argue that in your second case, there's no consistency "external" to the system; you've now defined a larger, distributed system that includes your external parties, and now you're in consistency-as-in-CAP territory, rather than consistency-as-in-ACID.
Re: Data consistency is overrated
#34I think there are two different kinds of consistency, and it's important to not conflate them. There's consistency that's internal to a system. Do all of the foreign keys line up correctly? Have I lost any data that was provided to me? Here, we can aspire to be 100% correct. I don't think the examples in this article conflict with that. Then there's consistency that's external to a system. This can be between this sy…
Hm, I'd cut the cake a little differently. I think there's two kinds of consistency we can strive for: - Strict consistency. Every pointer in my b-tree must point to a b-tree node, and not random data which could cause the program to crash. In a financial world, a bank should never print money. - Fuzzy "good enough" consistency. In the examples in the article, all of the financial transactions should end up close eno…
>In a financial world, a bank should never print money.
Except banks routinely do "print money" (if you agree that credit and money are effectively the same). Even without delving into workings of our financial system, there are enough stories of banks making mistakes which erroneously increase their liabilities to clients. Banks simply revert such mistakes if they can or sue clients if money were withdrawn before such mistake was found. So it's much closer to fuzzy/eventual consistency.
Re: Data consistency is overrated
#35I think there are two different kinds of consistency, and it's important to not conflate them. There's consistency that's internal to a system. Do all of the foreign keys line up correctly? Have I lost any data that was provided to me? Here, we can aspire to be 100% correct. I don't think the examples in this article conflict with that. Then there's consistency that's external to a system. This can be between this sy…
Yeah I tend to agree. Another way of looking at it is that consistency helps curb complexity, and complexity is our biggest nemesis. If breaking consistency is necessary, so be it. Sometimes it’s the only way forward, especially so for horizontal scalability. But at the same time, if we don’t have consistency within small units, we get circumstantial complexity for no good benefit.
As usual, balancing trade offs is better than following dogma, and within our current technological paradigm I don’t see any alternative but to minimize inconsistency for large distributed systems. Perhaps in the future we find something that eliminates a bunch of these problems altogether, but that’s not today.
Re: Data consistency is overrated
#36Sounds like an excuse.
Re: Data consistency is overrated
#37Earlier quoted context omitted.
There's a ton of places where foreign keys are used wrong - deletion is acceptable, and good error handling for that case is the right thing to build anyway. That's one of the key arguments that nonconsistency advocates are arguing for. If you build a music playlist, the behavior of the program if the mp3 files referenced within should be to skip that track, not to crash. I've heard too many people arguing that they…
Deletion is acceptable, and if you have everything in a consistent system they will both exist or not. "Nonconsistency advocacy" doesn't make a lot of sense to me here. Are you advocating not relying on consistency in systems that guarantee consistency? That is a waste of time. Are you advocating eschewing consistent systems? Well, then you have more work, so only if I need to. And yes, you should handle data errors…
My point is: Removing a music track from the list of playable tracks is not a good reason to remove it from playlists. Having dangling pointers there, and handling them appropriately is the right thing to do.
This is true in a ton of cases where neither soft-delete or cascading delete make sense. Rarely, if ever, have I actually encountered the latter.
Re: Data consistency is overrated
#38The problem with no consistency isn't the impact to users, but developers. If things can be inconsistent you need to worry about a whole new class of failure cases where data is inconsistent, and handle them correctly.
Part of this is guarantees about correctness. Is there a bug that manifests even without inconsistency, or was there just some weird inconsistent state that caused an issue? If you don't have consistency you have to ask this about every single bug, making debugging problems much harder.
Re: Data consistency is overrated
#39I think there are two different kinds of consistency, and it's important to not conflate them. There's consistency that's internal to a system. Do all of the foreign keys line up correctly? Have I lost any data that was provided to me? Here, we can aspire to be 100% correct. I don't think the examples in this article conflict with that. Then there's consistency that's external to a system. This can be between this sy…
Hm, I'd cut the cake a little differently. I think there's two kinds of consistency we can strive for: - Strict consistency. Every pointer in my b-tree must point to a b-tree node, and not random data which could cause the program to crash. In a financial world, a bank should never print money. - Fuzzy "good enough" consistency. In the examples in the article, all of the financial transactions should end up close eno…
> In a financial world, a bank should never print money.
It's not relevant to the point but this is absolutely what banks do in the UK and I expect in most modern economies. They create money.
https://www.bankofengland.co.uk/explainers/how-is-money-crea...
Re: Data consistency is overrated
#40Data consistency is overrated if your business is ok with that. Many businessed are not ok with that. Example: airline, booking process.
How about I order something, but the inventory was off, so I never get my item. This is how you get unhappy customers. You could have buffer inventory to help with your bad system, but then you are tying up money in inventory..