InfluxDB has taken its open-source business to Silicon Valley • Hacker News Board

Re: InfluxDB has taken its open-source business to Silicon Valley

#21

Post by gtrubetskoy » Wed, Mar 25, 2015, 7:07 PM UTC

pauldix wrote: post #20

gtrubetskoy wrote: post #19

Earlier quoted context omitted.

Thanks Paul! So you're saying it's all a SMOP :) Another thing that I think might be a critical (or at least interesting) characteristic is back-filling optimization, i.e. when you need to load a trillion data points of historical data - this y/t explains it pretty well and talks about how OpenTSDB addresses it: https://www.youtube.com/watch?v=SgD3RD2Shg4 Anyhow - keep up the good work, I very much believe that in th…

Cool, I'll have to take a look at that talk. We've had people ask about backfilling large amounts of data so it's something we'll have to figure out.

Another thing I was curious about is why not do all the clustering/distributed stuff at the db level, i.e. have some sort of a distributed BoltDB-like/Raft as a separate layer or even entirely separate project, and then InfluxDB would be a much thinner/simpler thing. I think that in general the approach of OpenTSDB and similar things is right, it's just that HBase/Hadoop is a such a pain to set up and maintain (and so is Cassandra, if perhaps a little less).

Re: InfluxDB has taken its open-source business to Silicon Valley

#22

Post by pauldix » Wed, Mar 25, 2015, 7:43 PM UTC

gtrubetskoy wrote: post #21

pauldix wrote: post #20

Earlier quoted context omitted.

Cool, I'll have to take a look at that talk. We've had people ask about backfilling large amounts of data so it's something we'll have to figure out.

Another thing I was curious about is why not do all the clustering/distributed stuff at the db level, i.e. have some sort of a distributed BoltDB-like/Raft as a separate layer or even entirely separate project, and then InfluxDB would be a much thinner/simpler thing. I think that in general the approach of OpenTSDB and similar things is right, it's just that HBase/Hadoop is a such a pain to set up and maintain (and s…

One of the key goals of the project is to be able to aggregate and downsample from raw high precision data. That means we want a framework in which we ship the code to where the data lives, not the other way around.

The abstractions I've seen that have the database layer and then some services on top all miss this. They transport all of the raw data over the network and then run the computations and return the summary ticks back to the user.

Our framework lets us compute the summary ticks locally and send only those back (is many cases, but not all).

Re: InfluxDB has taken its open-source business to Silicon Valley

#23

Post by iolco51 » Thu, Mar 26, 2015, 9:03 PM UTC

bad_user wrote: post #6

Any comparisons with KairosDB or OpenTSDB?

KairosDB is less popular, smaller, but with different limitations and is more flexible. It was inspired by the OpenTSDB design but then took a different path.

OpenTSDB relies on HBase, KairosDB has configurable and pluggable datastore - but the only production-ready so far is using Cassandra.

OpenTSDB always does interpolation of values for aggregation (which I found to be an hazardous decision), KairosDB does not really do proper series "vertical" aggregation (by vertical I mean not downsampling).

OpenTSDB is GPL, KairosDB is Apache 2.0 (that counts for closed-source integrations).

OpenTSDB supports only numerical data but supports annotations, KairosDB supports Strings and numerical in baseline but is compatible with any data type, it does not have annotations (but you may use string for that).

On their baseline OpenTSDB produces graphs on the server, KairosDB produces graphs on the client.

Both are integrated with Grafana time series dashboard, OpenTSDB has more side projects, KairosDB is the only time series database I know for being integrated with a reporting tool (BIRT).

OpenTSDB requires to create the metrics in advance using a special tool (needs to lock the cluster to allocate a new ID), KairosDB can have any kind of new metric on the fly.

If you need something modular for building custom features I strongly recommend KairosDB. Look at the code, it's really nicely crafted.

Otherwise, they both have goods, I found that KairosDB is also much less limited on the cornersides (while having less side-projects), and we now use kairosDB intensively.