Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cassandra is not row level consistent

datanerds.io

Post reply on HN
Print view View original

111–120 of 126 posts

Re: Cassandra is not row level consistent

#111

Post by brianwawok »

hamax wrote: post #110
tremon wrote: post #107

Earlier quoted context omitted.

If I follow your argument correctly, it is basically the same argument as "your C compiler is correct, what you've written is invalid and the standard allows undefined behaviour here". Which may be a technically valid argument against the compiler/database system, but it's not a valid argument for defending the system as a whole: if a standard allows arbitrary execution instead of bailing out on non-standard (ambiguo…

Is variable assignment in c/java/... unreliable? It behaves very similar to what C* does. Concurrent access and modification will produce undefined behaviour if you don't explicitly protect it.

Exactly.

Getting access to things like concurrent locks is HARD to get right. That is why there are so many simple languages that don't let you touch concurrency.

Doesn't mean there is no need for it in the world, and no one should be able to use it.

Re: Cassandra is not row level consistent

#112

Post by pkolaczk »

isoap wrote: post #101
ncw33 wrote: post #91

Earlier quoted context omitted.

I agree, Cassandra is doing here exactly what I (as a user) would expect. When using a DB like C*, you always have to ask yourself, "does this update happen before or after this one - have I done anything to ensure that's the case?" In this example, the second query (the UPDATE) is being partially applied before the first query (the INSERT) - and that's OK, because there's no ordering or dependency in the second quer…

> the second query (the UPDATE) is being partially applied before the first query (the INSERT) - and that's OK "Partially applied" is ok with a database? The description of Cassandra on it's site is "Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data." If it's mission-critical data, I wouldn't do arbitrary things with it f…

Cassandra is perfectly fine. If you want your writes to be consistent, learn to use LWTs properly. The same way, if you want your data to be fully consistent in RDBMS, learn to use SERIALIZABLE isolation level (which is not default in most RDBMSes for performance reasons). If, in an RDBMS, you use SERIALIZABLE for half of the updates and READ UNCOMMITTED for another half, guess what consistency guarantees do you really get?

Re: Cassandra is not row level consistent

#113

Post by pkolaczk »

brianwawok wrote: post #104
isoap wrote: post #101

Earlier quoted context omitted.

> the second query (the UPDATE) is being partially applied before the first query (the INSERT) - and that's OK "Partially applied" is ok with a database? The description of Cassandra on it's site is "Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data." If it's mission-critical data, I wouldn't do arbitrary things with it f…

This is not arbitrary. It may be not intuitive coming from a rdbms background, but it isn't arbitrary. As Johnathon pointed out, it's like properly using synchronized or volatile half the time. I don't call it sometimes not working as arbitrary. I call it expected for not following the rules of the system.

RDBMSes can cause similar inconsistencies if you don't know what you're doing. It is like setting read uncommitted and then complaining about dirty reads.

Re: Cassandra is not row level consistent

#114

Post by dpark »

SEJeff wrote: post #98
dpark wrote: post #52

Earlier quoted context omitted.

Did anyone else ever publish a meaningful response? I thought Antirez's response was reasonable but I also recognize the inherent bias. I'd love to see responses from other experts, ideally written such that non-experts can reasonably understand.

I recall several people on twitter (I follow a lot of distributed systems people / cs professors as it is part of my job to build these things) who agreed with Martin's analysis.

I'll have to see if I can dig some of the responses up. It bugs me that I still don't know if this algorithm is actually safe.

Re: Cassandra is not row level consistent

#115

Post by utobi »

jbellis wrote: post #59

Cassandra developer here. Lots of comments here about how Cassandra is AP so of course you get inconsistent (non-serializable) results. This is true, to a point. I'm firmly convinced that AP is a better way to build distributed systems for fault tolerance, performance, and simplicity. But it's incredibly useful to be able to "opt in" to CP for pieces of the application as needed. That's what Cassandra's lightweight t…

Disclaimer: I am working with the author

The problem is even with using LWTs for both updates you are running into the same problem, the only thing you gain is that one of the statements (either lock or release) wins and in this case it then only works because the lock uses a TTL and is removed after some time.

Also, two conflicting statements from the same thread going to the same node should be easily serializable for Cassandra - or at least be logged.

> Don't take shortcuts and you won't get burned

Since you are the Datastax CTO, maybe some alignment with the marketing team on how features are communicated to users might help users not getting burned? :D

Re: Cassandra is not row level consistent

#116

Post by garyrichardson »

ncw33 wrote: post #91
jbellis wrote: post #59

Cassandra developer here. Lots of comments here about how Cassandra is AP so of course you get inconsistent (non-serializable) results. This is true, to a point. I'm firmly convinced that AP is a better way to build distributed systems for fault tolerance, performance, and simplicity. But it's incredibly useful to be able to "opt in" to CP for pieces of the application as needed. That's what Cassandra's lightweight t…

I agree, Cassandra is doing here exactly what I (as a user) would expect. When using a DB like C*, you always have to ask yourself, "does this update happen before or after this one - have I done anything to ensure that's the case?" In this example, the second query (the UPDATE) is being partially applied before the first query (the INSERT) - and that's OK, because there's no ordering or dependency in the second quer…

My biggest complaint about C: SQL-like interface. These seems like a terrible decision to me.

If you are used to relational databases and SQL, it's a real struggle to get your head in the right place. Some of the gotchas:

Cell level vs row level consistency/locking (as stated in this article) * Grouping / counting isn't really a thing. * WHERE statements are actually used for identifying the key/value pair you want and not for filtering the rows. * Indexes aren't really indexes. They are definitions of partitions and sort order.

Re: Cassandra is not row level consistent

#117

Post by prodigal_erik »

hamax wrote: post #103
isoap wrote: post #102

Earlier quoted context omitted.

In such a situation though, shouldn't it raise an error? At least that way, the user could have a chance to recover. Why should it make an arbitrary decision that unknowingly corrupts data for some users?

The only way for C* to know that it should raise an error is if it would implicitly protect all writes with LWT and this is not what most users want. Following the parent's example, if you don't protect memory access with a lock, you can't know that somebody else locked it.

"Must be consistent" is a property of the data, not of a particular transaction. Data on which inconsistent transactions are ever allowed should be declared, preferably with big hazard signs.

Re: Cassandra is not row level consistent

#118

Post by j_s »

jbellis wrote: post #59

Cassandra developer here. Lots of comments here about how Cassandra is AP so of course you get inconsistent (non-serializable) results. This is true, to a point. I'm firmly convinced that AP is a better way to build distributed systems for fault tolerance, performance, and simplicity. But it's incredibly useful to be able to "opt in" to CP for pieces of the application as needed. That's what Cassandra's lightweight t…

The situation could be considered similar to .NET adding the IllegalCrossThreadException exception to Windows Forms in v2¹, defaulting to detecting invalid behavior with the option to disable checking (Control.CheckForIllegalCrossThreadCalls).

.NET v4.5 introduced a new approach (Task-based Asynchronous Pattern)² with changes to framework and language (C#5) to "simplify" implementing things correctly.

.NET took the developer-friendly path; Cassandra, not yet.

¹ https://web.archive.org/web/20060414185346/http://msdn.micro...

² http://stackoverflow.com/a/18033198

Re: Cassandra is not row level consistent

#119

Post by SEJeff »

dpark wrote: post #114
SEJeff wrote: post #98

Earlier quoted context omitted.

I recall several people on twitter (I follow a lot of distributed systems people / cs professors as it is part of my job to build these things) who agreed with Martin's analysis.

I'll have to see if I can dig some of the responses up. It bugs me that I still don't know if this algorithm is actually safe.

Honestly given Antirez's response I'm genuinely surprised no one has written a TLA+ or similar formal verification proof. Then the outcome is binary; either redlock is sound and works as expected, or it is not.

Amazon's AWS Architect, James Hamilton is a big fan of this approach, as are most of the heavyweights in distributed systems:

http://perspectives.mvdirona.com/2014/07/challenges-in-desig...

EDIT: prefaced URL with http

Re: Cassandra is not row level consistent

#120

Post by im_down_w_otp »

pkolaczk wrote: post #90
im_down_w_otp wrote: post #44

Earlier quoted context omitted.

Quorum won't solve consistency problems like this unless you can also guarantee a serialized reader/writer for any given piece of data. The best you can do by pinning queries to primary replicas and enforcing R + W > N is RYOW consistency, and that depends on the aforementioned serialization point.

Quorum reads + writes and applying non-destructive updates only are enough to get linearizabile consistency in Cassandra, even if clocks are not perfectly synchronized.

If you're applying non-destructive updates then you don't even need quorum to achieve consistency. You're just writing immutable data.
Post reply on HN