We use KDB for time series data but it is expensive.
He's anyone checked this project out? http://www.monetdb.org/Home
Cassandra vs MongoDB For Time Series Data
11–20 of 82 posts
Re: Cassandra vs MongoDB For Time Series Data
#12We use KDB for time series data but it is expensive. He's anyone checked this project out? http://www.monetdb.org/Home
The column store research that started it all. Definitely read the paper if you haven't already:
"The Researcher’s Guide to the Data Deluge: Querying a Scientific Database in Just a Few Seconds" (http://oai.cwi.nl/oai/asset/18546/18546B.pdf)
Re: Cassandra vs MongoDB For Time Series Data
#13One thing I always find interesting about these kinds of problems is that most DBs don't describe how they're implemented. It's easy to use the wrong tool, and then once you learn how e.g. Mongo is implemented, it's obvious, "oh, that's why things are slow". I'd love to see http://eagain.net/articles/git-for-computer-scientists/ , but for every DB technology.
Agreed. Seeing a nice, clean user-facing API tells you practically nothing about the performance characteristics and failure modes you can expect to see.
The best database engine I've encountered in this respect is SQLite. It has plenty of information about what design tradeoffs it makes, and why, e.g.:
http://sqlite.org/lockingv3.html
Re: Cassandra vs MongoDB For Time Series Data
#14i work on scada systems which usually come with a time series database built in so that operators can do some basic plotting. Often these products have a 10 or 15 year legacy. The scada vendors seem to be careful to avoid making the time series database and plotting tools which come with the HMI packages too powerful, as this might cut in to their sales of Historian type products. If it wasn't a commodity already, th…
Might be interested in Cube that square released a few years ago. http://corner.squareup.com/2011/09/cube.html
edit: cube, not cuba :)
Re: Cassandra vs MongoDB For Time Series Data
#15I didn't see the size of the cluster described, but watch out for having timestamps as your row keys, for you are going to have hot spots (all timestamps in one token range).
What is your replication factor and the size of the cluster?
This might be improved with vNodes, though I'm not sure how granular and automatic the subnodes are. If they are just an even range (e.g. 256 vnodes across the same 00-ff range), then you will have the same problem.
This is the major reason why Datastax pushes random ordered partitioning so much, it's easy to get into hot water with byte-ordered keys.
Re: Cassandra vs MongoDB For Time Series Data
#16We use KDB for time series data but it is expensive. He's anyone checked this project out? http://www.monetdb.org/Home
My go-to for any time-series data is a column-oriented database.
fyi, Cassandra is row-oriented.
Re: Cassandra vs MongoDB For Time Series Data
#17We had some major issues using MongoDB for time series data due to the write volume (real time sensor data). The solution for us was Riak, mainly because we never need to update a vector clock, leaving us without the need for conflict resolution (and last write wins is fine for sensor data).
Why wouldn't generating some hashed _id work? It would then scale for writes easily (or, as in current mongodb, you could use hash-based indexes). I'm just asking since Riak seemed much slower for me when I tried it.
Hashed _id's aren't the bottleneck. Disks are the bottleneck. MongoDB is seriously lacking in the compression department and bogs down our disks constantly.
Riak is not slow as long as you're not running your cluster on VPSs. At our scale Riak's performance has been significantly better than MongoDB's due to our heavy write load, and there are fewer issues with using big disks with somewhat larger seek times.
Re: Cassandra vs MongoDB For Time Series Data
#18One thing I always find interesting about these kinds of problems is that most DBs don't describe how they're implemented. It's easy to use the wrong tool, and then once you learn how e.g. Mongo is implemented, it's obvious, "oh, that's why things are slow". I'd love to see http://eagain.net/articles/git-for-computer-scientists/ , but for every DB technology.
A pattern I've seen on a lot of the negative Mongo articles has been people using it for things they probably shouldn't. I've yet to use it for any load, and am struggling to triangulate from all the articles I read on whether it does/doesn't scale efficiently. All the issues I have hit so far have been self-inflicted, it is still one of the best new technologies I've used in years - but is has taken a while to stop…
For what kind of application or workload is MongoDB superior to the alternatives?
Re: Cassandra vs MongoDB For Time Series Data
#19I didn't see the size of the cluster described, but watch out for having timestamps as your row keys, for you are going to have hot spots (all timestamps in one token range). What is your replication factor and the size of the cluster? This might be improved with vNodes, though I'm not sure how granular and automatic the subnodes are. If they are just an even range (e.g. 256 vnodes across the same 00-ff range), then…
You're right. As I mention in the article, the row keys are not timestamps, the columns are timestamps. We use the RandomPartitioner for rows.
Re: Cassandra vs MongoDB For Time Series Data
#20One thing I always find interesting about these kinds of problems is that most DBs don't describe how they're implemented. It's easy to use the wrong tool, and then once you learn how e.g. Mongo is implemented, it's obvious, "oh, that's why things are slow". I'd love to see http://eagain.net/articles/git-for-computer-scientists/ , but for every DB technology.
A pattern I've seen on a lot of the negative Mongo articles has been people using it for things they probably shouldn't. I've yet to use it for any load, and am struggling to triangulate from all the articles I read on whether it does/doesn't scale efficiently. All the issues I have hit so far have been self-inflicted, it is still one of the best new technologies I've used in years - but is has taken a while to stop…
Hopefully this article wasn't seen as bashing MongoDB. I think I mention it was pretty reliable for us even under load. We were just solving a problem with it for which it wasn't the best solution.