Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

InfluxDB vs. Cassandra for timeseries data

influxdata.com

Post reply on HN
Print view View original

11–20 of 34 posts

Re: InfluxDB vs. Cassandra for timeseries data

#11

Post by msiebuhr »

twa927 wrote: post #9
paulasmuth wrote: post #5

The linked article is an obviously bullshit benchmark that makes influxdb look good and cassandra look bad (by, surprise, the influxdb folks). I'm far from a cassandra fanboy, but this really is just dishonest marketing. Not sure if that will work if your product is open source and the target audience are developers. Some thoughts: - The reason why cassandra uses so much more space to store the same data is that they…

Thanks for the analysis of their benchmark, I wanted to view the details by myself but it required creating an account on their page. > There is no superior "compression" technology Isn't it feasible to employ special encoding for time series data? For example, to encode a series of timestamps like 1473333629, 1473333630, 1473333631 you could encode it as 1473333629, +1, +2 (where +1, +2 are encoded in one byte). And…

Facebook does this (and quite a few other tricks) for storing time-series data in Gorilla (in-memory TSDB, Paper: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf), getting to 1,37 B per sample.

Prometheus implemented the Gorilla-bits (see https://prometheus.io/blog/2016/05/08/when-to-use-varbit-chu...) and reports getting down to 1,28 B per sample on some workloads, though at a cost of increased query-latencies.

Re: InfluxDB vs. Cassandra for timeseries data

#12

Post by deluvas »

dz0ny wrote: post #2

It would be interesting to compare memory requirements, I chose Influxdb because it had 10 times lower memory usage. The dataset was small (couple of million datapoints)... but stil

How much memory are we talking about? How often did you execute queries on the data?

I'm asking because my first impression of Influxdb involved lots of memory gobbling.

Re: InfluxDB vs. Cassandra for timeseries data

#13

Post by pauldix »

paulasmuth wrote: post #5

The linked article is an obviously bullshit benchmark that makes influxdb look good and cassandra look bad (by, surprise, the influxdb folks). I'm far from a cassandra fanboy, but this really is just dishonest marketing. Not sure if that will work if your product is open source and the target audience are developers. Some thoughts: - The reason why cassandra uses so much more space to store the same data is that they…

The tests were run on the same hardware, a single server. Bare metal, not VMs. InfluxDB writes the series string with everything. We tried to imitate what you'd need to do to get close to similar functionality doing time series like InfluxDB does in Cassandra.

If you're just going to write a bunch of uint64 keys with float64 values, of course Cassandra will get much faster. It would be trivial to make a time series database that outperforms InfluxDB with those limitations as well.

The point of the comparison is that InfluxDB gives you a ton of functionality out of the box and has great performance.

Again, the point is that if you want to do time series on Cassandra, you're going to write a bunch of the code yourself.

Re: InfluxDB vs. Cassandra for timeseries data

#14

Post by paulasmuth »

pauldix wrote: post #13
paulasmuth wrote: post #5

The linked article is an obviously bullshit benchmark that makes influxdb look good and cassandra look bad (by, surprise, the influxdb folks). I'm far from a cassandra fanboy, but this really is just dishonest marketing. Not sure if that will work if your product is open source and the target audience are developers. Some thoughts: - The reason why cassandra uses so much more space to store the same data is that they…

The tests were run on the same hardware, a single server. Bare metal, not VMs. InfluxDB writes the series string with everything. We tried to imitate what you'd need to do to get close to similar functionality doing time series like InfluxDB does in Cassandra. If you're just going to write a bunch of uint64 keys with float64 values, of course Cassandra will get much faster. It would be trivial to make a time series d…

> The point of the comparison is that InfluxDB gives you a ton of functionality out of the box and has great performance. [...] if you want to do time series on Cassandra, you're going to write a bunch of the code yourself.

Fair enough. I'm sure InfluxDB is very good/fast at timeseries data (allthough I have to admit to not actually having tried it out so far). Still, if that was your point, consider removing these statements from the blog.

> InfluxDB outperformed Cassandra by 4.5x when it came to data ingestion.

> InfluxDB outperformed Cassandra by delivering 10.8x better compression.

> InfluxDB outperformed Cassandra by delivering up to 168x better query performance.

I think it would help make the point and not put the reader in a defensive position (when the statements are clearly not based on a fair comparison of the two products and will not hold under most conditions). Just my two cents.

Re: InfluxDB vs. Cassandra for timeseries data

#15

Post by michaelcampbell »

paulasmuth wrote: post #10
twa927 wrote: post #9

Earlier quoted context omitted.

Thanks for the analysis of their benchmark, I wanted to view the details by myself but it required creating an account on their page. > There is no superior "compression" technology Isn't it feasible to employ special encoding for time series data? For example, to encode a series of timestamps like 1473333629, 1473333630, 1473333631 you could encode it as 1473333629, +1, +2 (where +1, +2 are encoded in one byte). And…

Yes, the delta encoding scheme you described (and other fancy coding schemes such as bitpacking, varints, RLE or a combination thereof) are frequently employed in columnar storage formats and databases. Columnar storage is basically a generalization that allows one to apply these optimizations to all kinds of data (not just timeseries). One popular open-source implementation of columnar storage that I am not affiliat…

A commercial DB that (also) does this HP Vertica. They tout a 4:1 to 5:1 compression ratio on average; due to the nature of the data the firm I work for stores in it, we get quite a bit better than that. Delta encoding is just one of maybe 5 different schemes it can use for a given column.

Re: InfluxDB vs. Cassandra for timeseries data

#17

Post by coredog64 »

pauldix wrote: post #13
paulasmuth wrote: post #5

The linked article is an obviously bullshit benchmark that makes influxdb look good and cassandra look bad (by, surprise, the influxdb folks). I'm far from a cassandra fanboy, but this really is just dishonest marketing. Not sure if that will work if your product is open source and the target audience are developers. Some thoughts: - The reason why cassandra uses so much more space to store the same data is that they…

The tests were run on the same hardware, a single server. Bare metal, not VMs. InfluxDB writes the series string with everything. We tried to imitate what you'd need to do to get close to similar functionality doing time series like InfluxDB does in Cassandra. If you're just going to write a bunch of uint64 keys with float64 values, of course Cassandra will get much faster. It would be trivial to make a time series d…

Hasn't that work already been done? Cyanite and KairosDB both plug in to the broader Graphite ecosystem (more or less) and use Cassandra as a data store.

Time series data has also been a particular focus in the Cassandra community. DTCS was too complicated, so they came up with the easier and faster TWCS. I don't think this is on you, but I'd love to see a comparison with the latest stable 3.x and a multiple node cluster.

Re: InfluxDB vs. Cassandra for timeseries data

#18

Post by paulasmuth »

michaelcampbell wrote: post #15
paulasmuth wrote: post #10

Earlier quoted context omitted.

Yes, the delta encoding scheme you described (and other fancy coding schemes such as bitpacking, varints, RLE or a combination thereof) are frequently employed in columnar storage formats and databases. Columnar storage is basically a generalization that allows one to apply these optimizations to all kinds of data (not just timeseries). One popular open-source implementation of columnar storage that I am not affiliat…

A commercial DB that (also) does this HP Vertica. They tout a 4:1 to 5:1 compression ratio on average; due to the nature of the data the firm I work for stores in it, we get quite a bit better than that. Delta encoding is just one of maybe 5 different schemes it can use for a given column.

Just so sad that Vertica is proprietary so we can't see how they did it! ;)

On a serious note: Please check out EventQL [0] some time. It's very similar to Vertica in some ways and completely open-source. It's a new project (beta) and not nearly as mature as vertica yet though (still a long way to go).

[0] https://eventql.io/

Re: InfluxDB vs. Cassandra for timeseries data

#19

Post by soundoflight »

From using InfluxDB (up to v0.10 I think it was), it's a great database but performance REALLY depends on the cardinality of your data.

I can't stress it enough, calculate your cardinality before switching over to it. If your cardinality looks good, InfluxDB is a perfect, logical choice. I really enjoyed it and it is dirt simple to figure out. We had a junior dev just out of college with little experience set it up and get a high level of proficiency in a matter of hours.

Edit: I should point out, I was doing about 10 million records on my db (hosted on a Mac Mini in development!) a day with a 2 week sliding window. I was pushing the data from InfluxDB into custom D3 visualizations. I would cache certain queries in Redis, so I wasn't always hitting InfluxDB with each read request.

Re: InfluxDB vs. Cassandra for timeseries data

#20

Post by pauldix »

coredog64 wrote: post #17
pauldix wrote: post #13

Earlier quoted context omitted.

The tests were run on the same hardware, a single server. Bare metal, not VMs. InfluxDB writes the series string with everything. We tried to imitate what you'd need to do to get close to similar functionality doing time series like InfluxDB does in Cassandra. If you're just going to write a bunch of uint64 keys with float64 values, of course Cassandra will get much faster. It would be trivial to make a time series d…

Hasn't that work already been done? Cyanite and KairosDB both plug in to the broader Graphite ecosystem (more or less) and use Cassandra as a data store. Time series data has also been a particular focus in the Cassandra community. DTCS was too complicated, so they came up with the easier and faster TWCS. I don't think this is on you, but I'd love to see a comparison with the latest stable 3.x and a multiple node clu…

We'll be doing comparisons against Kairos and OpenTSDB in the coming months. We just get asked about Cassandra specifically quite a bit.
Post reply on HN