Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost every Cassandra feature has some surprising behavior

blog.parsely.com

Post reply on HN
Print view View original

51–60 of 79 posts

Re: Almost every Cassandra feature has some surprising behavior

#51

Post by fweespeech »

lobster_johnson wrote: post #48
dvirsky wrote: post #14

Earlier quoted context omitted.

We've moved most of our data to redis (which we were using way before cassandra), with a bit of classic MySQL for off-line data, and Redshift for analytics. Right now redis is not using cluster mode, but we'll switch to it eventually. It's a big move, but I find redis more predictable, and for our data sizes (10s of Gs), cheaper to maintain.

Do you use Redis as the master data store? Are you using Redis Cluster, or unclustered? My concern with Redis is reliability. Redis Cluster has problems with consistency [1], whereas unclustered Redis -- well, it syncs to disk every 10 seconds or so, but even then I'm concerned that the reliability of its on-disk structures haven't been as battle-tested as, say, PostgreSQL. Or has it? [1] https://aphyr.com/posts/307-…

> well, it syncs to disk every 10 seconds or so

Yeah, about that:

http://redis.io/topics/persistence

> Using AOF Redis is much more durable: you can have different fsync policies: no fsync at all, fsync every second, fsync at every query. With the default policy of fsync every second write performances are still great (fsync is performed using a background thread and the main thread will try hard to perform writes when no fsync is in progress.) but you can only lose one second worth of writes.

Re: Almost every Cassandra feature has some surprising behavior

#52

Post by fweespeech »

manishsharan wrote: post #20
dvirsky wrote: post #14

Earlier quoted context omitted.

We've moved most of our data to redis (which we were using way before cassandra), with a bit of classic MySQL for off-line data, and Redshift for analytics. Right now redis is not using cluster mode, but we'll switch to it eventually. It's a big move, but I find redis more predictable, and for our data sizes (10s of Gs), cheaper to maintain.

If you moved your data off Caasandra to Redis, I am going to assume you must be using a lot of Lua to maintain some relationships in data in Redis. How is the performance? I have been experimenting with Redis and my lua scripts are long and I have been wondering how they will fare under production level loads.

A Redis Hash is similar enough to Casandra's data structure I doubt they'd need that.

row key = key

column = field

I know Lua is "an option" in Redis but I avoid it like the plague because under production loads, Redis hashes perform and are easy to maintain w/o Lua.

Re: Almost every Cassandra feature has some surprising behavior

#53

Post by lobster_johnson »

manishsharan wrote: post #19

Cassandra or most NoSQL databases seem to require a lot more internals knowledge than what a development or an ops team would have. Most sysops teams have a cadre of certified DBAs and administrators and they get on by without any surprises. The one thing Oracle got right was their certification program which ensures there are no gotchas -- most DBA can deliver smooth operations. I guess it is because these NOSQL pro…

On the other hand, if you tried to fit the article's use case -- at that scale -- into something like PostgreSQL you would still have to start learning the internals.

I think the article author's problem as that they assumed Cassandra would be similar to the technology they were already familiar with, which was not the case; it's different in so many areas, especially with regard to its performance profile, and to someone used to relational databases it's outright alien.

At the scale they're describing, they do exceed the threshold point at which one has to learn the internals of the technology. In engineering, scale is everything. Riding a bicycle doesn't require that you know how bicycles work, but sending a rocket to the moon demands a lot of knowledge about a wide range of subjects.

Re: Almost every Cassandra feature has some surprising behavior

#54

Post by lobster_johnson »

fweespeech wrote: post #51
lobster_johnson wrote: post #48

Earlier quoted context omitted.

Do you use Redis as the master data store? Are you using Redis Cluster, or unclustered? My concern with Redis is reliability. Redis Cluster has problems with consistency [1], whereas unclustered Redis -- well, it syncs to disk every 10 seconds or so, but even then I'm concerned that the reliability of its on-disk structures haven't been as battle-tested as, say, PostgreSQL. Or has it? [1] https://aphyr.com/posts/307-…

> well, it syncs to disk every 10 seconds or so Yeah, about that: http://redis.io/topics/persistence > Using AOF Redis is much more durable: you can have different fsync policies: no fsync at all, fsync every second, fsync at every query. With the default policy of fsync every second write performances are still great (fsync is performed using a background thread and the main thread will try hard to perform writes wh…

RDB is what I was thinking of. I didn't know they had added a write-ahead log. It seems you can combine RDB and AOF, which is nice.

Re: Almost every Cassandra feature has some surprising behavior

#55

Post by schmichael »

cassandranot wrote: post #26
dvirsky wrote: post #5

The thing is that contrary to many other tools, you can't run Cassandra with almost any of its default settings (maybe besides ports). In most tools you'll need to tweak a few defaults, with Cassandra you need to thoroughly read the docs on every little configuration in the server and schema definitions (especially if you're working in multiple DCs), you'll always find a little surprise if you skim through it. It's a…

We found cassandra to be too finicky and flakey and ended up using mongodb - simple to get going, admin, use and maintain. Cassandra was a huge pain in the ass, had to read all the source code just to get it running!

> We found cassandra to be too finicky and flakey and ended up using mongodb

Oh my. I look forward to your next "X is not a panacea" article on MongoDB.

Turns out commercial db vendors lie. A lot. Never use a feature until it's been in a couple point releases (y in x.y.z versioning). Upgrade slowly.

Also, Cassandra - and MongoDB when used with clustering - requires writing your data to best fit your queries. This often leads to wildly different schemas than you'd create in a traditional relational database.

You should probably read Kyle Kingsbury's article on MongoDB to have some idea of what you're really getting into: https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-read...

Re: Almost every Cassandra feature has some surprising behavior

#56

Post by fweespeech »

lobster_johnson wrote: post #54
fweespeech wrote: post #51

Earlier quoted context omitted.

> well, it syncs to disk every 10 seconds or so Yeah, about that: http://redis.io/topics/persistence > Using AOF Redis is much more durable: you can have different fsync policies: no fsync at all, fsync every second, fsync at every query. With the default policy of fsync every second write performances are still great (fsync is performed using a background thread and the main thread will try hard to perform writes wh…

RDB is what I was thinking of. I didn't know they had added a write-ahead log. It seems you can combine RDB and AOF, which is nice.

Yeah, you can. Its quite nice, the only real problem with it is the reliability of the clustering is subpar.

If you are happy with a Master/Slave setup and occasionally having to deal with data loss due to Master failures [e.g. losing a second or two of data that wasn't replicated], it works nicely.

Just realize I wouldn't use Redis as a long term persisting of data because of issues like this:

https://muut.com/blog/news/april-2014-service-failure.html

Re: Almost every Cassandra feature has some surprising behavior

#57

Post by IndianAstronaut »

pixelmonkey wrote: post #23

OP here. Surprised to find this article on the HN front page, a month after we originally posted it. Glad to answer questions. Ask me anything!

Are you using Cassandra for analysis? What is your analysis workflow like?

Re: Almost every Cassandra feature has some surprising behavior

#58

Post by cdelsolar »

easytiger wrote: post #16
threeseed wrote: post #12

Earlier quoted context omitted.

I have to disagree. But again given how scalable Cassandra is we could live in different worlds. I've run multi-gigabyte data sets in Cassandra without any config changes. And when I responsible for a 40 node Cassandra cluster we didn't do any tweaks other than a few JVM settings here and there. Cassandra I have to admit though is very sensitive to how you model and store your database. The whole tombstone saga is ne…

> I've run multi-gigabyte data sets WOW. You've run several GB of data through it? Wow.

multi != several

Re: Almost every Cassandra feature has some surprising behavior

#59

Post by samkone »

robusto wrote: post #37
samkone wrote: post #32

Earlier quoted context omitted.

Here's laughing. I guess you chose convenience over performance or scalability, which is a valid choice. But We're running a 1000 Cassandra nodes to handle Millions of ops where mongo would simply require too much admin work. the whole clusters are managed by two guys. Mongo's simplicity comes at a cost at some point.

What are you using for managing those 1000 nodes? opscenter or something else?

Yes opscenter, netlfix's priam, Cassandra cluster manager and the usual monitoring stuff, Datadog in our case. So far so good.

Re: Almost every Cassandra feature has some surprising behavior

#60

Post by ufo »

Tloewald wrote: post #49

The thing about encoding multiple fields within a cell using \x01 somewhat bugs me -- not because it's a hack but because it's yet another example of needless reinvention of wheels. Good old ASCII has characters specifically devoted to separating fields, keys, etc. that no-one uses for anything else. Why not use them instead of inventing a new character that does the same thing? (By the same token, there was never an…

as a younger programmer, the only place I have ever seen these characters is that the fish shell uses the Record Separator character when it exports an environment variable that is an array.
Post reply on HN