51–60 of 73 posts
Below are the tweets by Aphyr for the same thing (with language slightly toned down). I never thought I won't find their mention here on HN. :) Nevertheless, precise, to the point: Buncha people giving me for calling data written through an invariant violation "corrupted state", like somehow it's not garbage. If TCP checksums don't work right we don't call the packet "inconsistent." We call it corrupt. If a disk shuf…
Earlier quoted context omitted.
oh, but it is an important difference, i believe. any experienced developer will keep an audit log of transactions somewhere. a corrupted database can be completely lost. an inconsistent one can be later corrected by using the audit trail. corruption is a really scary word to hear about a database. inconsistency is also bad, but corruption is worse, IMO. P.S. but, i must add - a DB that says it's isolated, but really…
> any experienced developer will keep an audit log of transactions somewhere. a corrupted database can be completely lost. an inconsistent one can be later corrected by using the audit trail. If you have a full audit trail you can rebuild the database from scratch so there is no difference between an inconsistent and a corrupted database. In fact, since the audit trail ought store responses from the database, replayi…
for starters, i was not thinking about blindly replaying it in the same system. of course that makes no sense. rather, after you find the bug, you use an alternate source of the same data to reconstruct the correct state AFTER you have fixed the bug. e.g. i had a case, where one table held state of a stock, while other held documents which were needed as proof of changes of the state. you can typically never just change things, this is not only a programmer practice, this is a standard accounting practice as well. had in my example the database gotten corrupt, that would be it [1]. but it was not, and i was able to, after finding the bug, locate exactly when and where it had happened, and correct it's effects as well.
of course, this type of save will often not be possible, but i just wanted to point out that it is sometimes possible, and makes a difference. ultimately, it's just not the same. and IMO, the guy writing the call me maybe blog should not bitch about it. it's a minor, honest mistake, that can in no way reflect badly on his reliably great work.
[1] while writing this, another thought came to mind - had it gotten corrupt, we would have used a backup :D not the end of the world also. but i think the distinction somewhat matters. a corrupt database is a headache for everyone, and freaks people out. business people too.
Earlier quoted context omitted.
oh, but it is an important difference, i believe. any experienced developer will keep an audit log of transactions somewhere. a corrupted database can be completely lost. an inconsistent one can be later corrected by using the audit trail. corruption is a really scary word to hear about a database. inconsistency is also bad, but corruption is worse, IMO. P.S. but, i must add - a DB that says it's isolated, but really…
The assumption that you can "fix" the database and proceed as if nothing had happened is naive, IMNSHO. Firstly, there's usually non-trivial amount of time until the inconsistency is discovered, that one modification may be easily followed by many additional modifications that are not simply commutative and "diffuse" the inconsistency to other records. For example if we're talking about bank accounts, you often get i…
but yes, the response here is bad.
Am I missing something or does this not address the main issue the original article raised: The documentation is simply incorrect. It claims to support SNAPSHOT ISOLATION but does not. The company knows this and even this article says the behaviour "is totally expected". Seems like the first response should be to fix the docs and not claim capabilities beyond what's implemented. (Also it was pretty clear from the ori…
Disclaimer: I work for Percona. The docs are on the galeracluster.com page, which is owned and maintained by another company, so there's no way we could fix those. A staff member from this company (And one of the Galera authors) replied on the original 'Call me maybe' post indicating they would fix the docs, though. I think the 'corruption vs inconsistency' debate could seem as nitpicking, but anybody who has been wo…
The article implies that inconsistent is better than corrupt, and that mixing up the terms paints a worse picture. The assumption is flawed. Inconsistent is as bad or worse than corrupted. It is a silent failure, and silent failures are worse than visible ones.
Am I missing something or does this not address the main issue the original article raised: The documentation is simply incorrect. It claims to support SNAPSHOT ISOLATION but does not. The company knows this and even this article says the behaviour "is totally expected". Seems like the first response should be to fix the docs and not claim capabilities beyond what's implemented. (Also it was pretty clear from the ori…
Disclaimer: I work for Percona. The docs are on the galeracluster.com page, which is owned and maintained by another company, so there's no way we could fix those. A staff member from this company (And one of the Galera authors) replied on the original 'Call me maybe' post indicating they would fix the docs, though. I think the 'corruption vs inconsistency' debate could seem as nitpicking, but anybody who has been wo…
I'd argue that "corrupted" is usually used in cases when the data get messed up due to external circumstances (say, disk error corrupting a page of data) or because of a clear bug (say, writing to incorrect place), while "consistency" is usually used for data violating some invariants (foreign keys, check constraints, unique constraints, or global constraints like "total sum of money is X").
But in the end this is utterly irrelevant because the database failed to provide correct data (to the extend implied by the advertised guarantees like isolation level, for example). For the user the end result is pretty much the same.
And actually, corrupted data tend to be detected much sooner, because when you can't read the disk pace you know there's an error and can address it. The inconsistency is often creepy and takes very long until someone spots that.
Earlier quoted context omitted.
I think the Percona writer is focusing on "corruption" in terms of how I think most database folks imagine "corruption" - where a series of commands will erase a block of data, or make a block of data unrecoverable. I agree that "inconsistent" is probably more mechanically accurate, but since in this case the side effect of the inconsistency is that you can't trust the contents of a block of data, it seems like a dif…
Yup, inconsistency is a kind of data corruption, provided you expect it to be in a consistent state. The emotional value of the latter is much stronger, though, and it sounds worse to the uninitiated, which is probably why Percona folks are trying to spin it that way.
I know that isn't accurate, but the tendency for it to be true, actually makes the word inconsistent upset me more than corruption. I know how to fix bad data, trying to track down a soft error that is hard to reproduce is the subject of my nightmares.
Earlier quoted context omitted.
To some extent, isn't that the whole point of specifying the transaction isolation level you need? So you can make these assumptions?
Sure, but there are tradeoffs in performance then. I'm not saying transactions are bad (although maybe my strong wording in the parent implies that), just that, especially if you're using an ORM, you shouldn't make those assumptions by default.
Not very classy. And the point of Aphyr still stands I think. In default mode it is easy to get corrupt data with Galera Cluster. That InnoDB on a single instance can have the same problem makes it all the more troubling and I'm glad I moved away from MySQL a long time ago.
> If you use this in a real life, the more obvious way to write these transactions is: Is it? Do ORMs really do that, or is it one of those "SQL was designed to be used this way, but nobody using SQL read the design documents" cases?
> Do ORMs really do that I haven't seen any, but they do use optimistic locking.
ORMs might issue lock hints but in general they simply start the transaction with the desired isolation level and let the RDBMS take care of the locking of affected rows/tables by statements executed.
Lock hints are sometimes used by naive ORMs who think you can implement optimistic concurrency using row locking. Their users will find out eventually that this whole row-locking doesn't help anyone as it simply reverses the tables on 'first write wins' vs. 'last write wins': there's still someone who loses, so the locking doesn't solve the problem of no-one losing their changes while it does give slower performance (and on SQL Server even the risk of deadlocks).
What developers often overlook is that there are two types of transactions: business transactions and DB transactions. A business transaction can span multiple DB transactions and a DB transaction can span multiple SQL statements. They're not the same, seeing a business transaction as equal to a DB transaction makes things get messed up and often gives food to the need of explicit lock hints for some queries to e.g. get the false sense of being able to implement optimistic concurrency. If you consider a business transaction, locking doesn't make any sense: it can take some time to complete it, so you have to deal with the side effects of stale data: it immediately becomes apparent that e.g. optimistic concurrency has no place here, one needs other ways to avoid people overwriting work of each other.
TFA re-orders statements to get the desired locks in place, and it IS possible to do so, e.g. some ORMs offer when to start a transaction or implement Unit of works which allow you to specify which batches to execute first (e.g. first deletes, then inserts). However the developer using the ORM isn't working at that abstraction level, as the ORM offers an abstraction level above all that; so re-ordering statements to get the desired read locks or avoid dirty reads within a transaction is abusing knowledge of how the abstraction offered by the ORM internally works. IMHO one shouldn't do that: the code utilizing the ORM is db agnostic: expecting certain DB behavior in DB agnostic code is going to give unpleasant side effects when the DB agnostic code will be used to e.g. utilize another RDBMS instead of the current one.
Earlier quoted context omitted.
Disclaimer: I work for Percona. The docs are on the galeracluster.com page, which is owned and maintained by another company, so there's no way we could fix those. A staff member from this company (And one of the Galera authors) replied on the original 'Call me maybe' post indicating they would fix the docs, though. I think the 'corruption vs inconsistency' debate could seem as nitpicking, but anybody who has been wo…
Corruption is read by the database people as: "Your data is unreadable"; Inconsistency as: "Your data is wrong". The article implies that inconsistent is better than corrupt, and that mixing up the terms paints a worse picture. The assumption is flawed. Inconsistent is as bad or worse than corrupted. It is a silent failure, and silent failures are worse than visible ones.
inconsistency is horrible, and theoretically the same thing, you are right! but it's something you might be able to fix, without the whole system failing, with maybe just a limited number of people knowing what happened and not the whole company and/or all customers. this is kinda fucked up, but i think it counts. (and PS, accountants find these errors. it's their job, and they have experience enough not to trust any proof of correctness, and check everything twice.)