OpenTSDB – A Distributed, Scalable Time Series Database

Disclosure: I work for Axibase.

If you have a large team of engineers writing data into it, they will sometimes abuse the schema by overloading single metrics with many tags. Every permutation of tag*value creates a new row per hour of data. This creates extremely hot shards when engineers inevitably do something like store a client IP address into a tag. When this happens, you may have to write some web-UI scraping code to figure out which regions are getting the most traffic (assuming you have "extremely high traffic setup" numbers of region servers), or script up something like misra-gries on a tcpdump of its inbound metrics to see where the hot shit is so you can get somebody's deploy reverted. I know of some companies that have forked OpenTSDB and prefixed each row with a hash such that it spreads overloaded metrics around the cluster much more evenly. KairosDB solves this by not using a lexicographically sharded database (Cassandra).

When run properly, it runs great. But that takes some really painful learning experiences to learn how to do, generally.

The hard part is HBase and to a lesser degree HDFS.. there are just a lot of things to read and learn about, and failure to address one of them will eventually cause interesting struggles. That said, my company looked at the field in 2012 and chose OpenTSDB, and it's handled things well enough considering fairly poor configuration of HBase. We recently reevaluate the space and tried Influx and Karios. Influx fell on its face in several ways that will take years to fully iron out, Kairos trades one set of evil for another, so we decided to do some learning and redeploy OpenTSDB correctly. As far as open source options, it's far ahead in the battle hardened department.

If you want to consider it, I would advise checking out some of the slides/videos on http://www.hbasecon.com/ - these two being the most critical:

* OpenTSDB and AsyncHBase Update

* HBase Performance Tuning @ Salesforce

HBase seems pretty vibrant. There's been a lot of progress with things like WAL shipping and performance tweaks like SSD block caching, off GC heap support, read access from multiple regionservers. OpenTSDB plugs along at a slower pace, but it's much more professional than i.e. Influx changing storage engines 5 time and getting worse with each. The OpenTSDB 2.2 release looks really exciting and fixes some hard high rate problems for both reads and writes: http://opentsdb.net/docs/build/html/new.html#id1

In the commercial space, Circonus looks pretty interesting and comes built and recommended by people I trust. It goes a lot farther than metrics storage. I wasn't able to get my management to do more than a video call to really dig in and evaluate it myself.