11–20 of 72 posts
Durability is also overrated. Jokes aside, I’ve been responsible for a system that processed ~1 billion monetary transactions per day. Even with a fanatical focus on consistency and correctness it was still always off. With the philosophy promoted on the OP the chaos would have been complete… is my gut feeling at least.
Data consistency is overrated if your business is ok with that. Many businessed are not ok with that. Example: airline, booking process.
I once worked on a platform producing analytics using data that, at its source, was manually typed in by people. My product managers would insist we do distinct counts on the aggregates instead of using probabilistic algorithms, because we "needed" the absolute 100% accurate output. No matter how many times I would explain the data was never 100% accurate to begin with and that the error rate using HyperLogLog wouldn…
There are benefits to simplicity. If there isn't a reason to need 100x faster performance, then why complicate things? That would be my why.
Data consistency is overrated if your business is ok with that. Many businessed are not ok with that. Example: airline, booking process.
Bad examples. Airlines are notorious for having incoherent booking policies. Over selling flights and such.
To the contrary, consistency is extremely important so that they overbook by exactly the right amount, to compensate for the statistically expected no-shows.
Durability is also overrated. Jokes aside, I’ve been responsible for a system that processed ~1 billion monetary transactions per day. Even with a fanatical focus on consistency and correctness it was still always off. With the philosophy promoted on the OP the chaos would have been complete… is my gut feeling at least.
Out of interest, what system processed 1 billion monetary transactions a day?
I once worked on a platform producing analytics using data that, at its source, was manually typed in by people. My product managers would insist we do distinct counts on the aggregates instead of using probabilistic algorithms, because we "needed" the absolute 100% accurate output. No matter how many times I would explain the data was never 100% accurate to begin with and that the error rate using HyperLogLog wouldn…
- They don't care about performance and think their users don't care about performance.
- They have sold something that doesn't make sense but sounds good.
- They are afraid of their users.
- Their users don't understand the problem you're describing or simply don't believe it.
- The business domain of their users actually knows the about the issue, but they still want or need the exact counts.
Durability is also overrated. Jokes aside, I’ve been responsible for a system that processed ~1 billion monetary transactions per day. Even with a fanatical focus on consistency and correctness it was still always off. With the philosophy promoted on the OP the chaos would have been complete… is my gut feeling at least.
Out of interest, what system processed 1 billion monetary transactions a day?
I think there are two different kinds of consistency, and it's important to not conflate them. There's consistency that's internal to a system. Do all of the foreign keys line up correctly? Have I lost any data that was provided to me? Here, we can aspire to be 100% correct. I don't think the examples in this article conflict with that. Then there's consistency that's external to a system. This can be between this sy…
That's one of the key arguments that nonconsistency advocates are arguing for. If you build a music playlist, the behavior of the program if the mp3 files referenced within should be to skip that track, not to crash. I've heard too many people arguing that they should be allowed to emit nasal demons if there's a data error, and they're wrong even in many well structured and tightly integrated datasets, but especially wrong in "web-scale" datasets
Earlier quoted context omitted.
Bad examples. Airlines are notorious for having incoherent booking policies. Over selling flights and such.
Overbooking is intentional policy. It has nothing to do with data consistency. To the contrary, consistency is extremely important so that they overbook by exactly the right amount, to compensate for the statistically expected no-shows.
And there is no "exactly right amount" that makes it work. They keep options for forcing people off flights if they planned it wrong.
Is also why they don't let gate agents over sell a flight. They keep a stronger consistency on that, for this exact reason. Over selling would fit in what someone else called external consistency.