Thoughts on Time-series Databases • Hacker News Board

Re: Thoughts on Time-series Databases

#121

Post by mjibson » Wed, Jul 01, 2015, 4:14 PM UTC

pauldix wrote: post #119

KyleBrandt wrote: post #116

At Stack Exchange our monitoring system bosun ( http://bosun.org ) can use different time series databases as long as they can be bent into tag key+tag value models. Currently it works best with OpenTSDB, but can also support graphite (and elasticsearch populated by logstash). InfluxDB query support is in a branch, but don't want to merge until we have a devoted Bosun+InfluxDB maintainer since we don't use it at Stac…

InfluxDB CEO here. Those problems with corrupting data were with the 0.8 line of releases. But to be honest there are people that have been running that and 0.7 in production for almost a year without problems. Your mileage may vary, but we're not supporting any releases prior to the 0.9 line. For the 0.9 set of releases, this is what we're supporting going forward. There are some queries that cause the server to cra…

I performed some OpenTSDB vs InfluxDB comparisons and found that InfluxDB used almost 20x the storage space and was 3x slower than OpenTSDB for an identical data set. The speed isn't that big of a problem and I'm convinced will get faster (esp with the write improvements in 0.9.1), but the space issue is harder to swallow.

HBase, which is what backs OpenTSDB, can compress data using LZO or snappy. Uncompressed, our test data was at 4GB, but went down to about 400MB after HBase compressed it. InfluxDB was using 8GB. OpenTSDB has done a lot of work to be byte efficient, and it's paid off. We hope InfluxDB will get to a similar place.

Re: Thoughts on Time-series Databases

#122

Post by MichaelGG » Wed, Jul 01, 2015, 5:06 PM UTC

rdtsc wrote: post #114

MichaelGG wrote: post #63

Earlier quoted context omitted.

What industry or app was this for? I wrote something that sounds like this for VoIP packet capture. Though the rates were around 5TB/day. I'd keep the latest indexes in RAM before delta-encoding and writing out. Periodically, merge multiple indexes into one. Worked surprisingly well, and the indexing overhead was only a few bits per packet (given enough similarity in packets over a short chunk of time (say, 1 minute)…

VoIP capture was one of the functions in the system too. But it had others. It looks like your implementation is more sophisticated, I didn't do any delta-encoding. Just periodic fsyncs. The maximum we tested was up to 1TB/day. Main index ( [(timestamp,offset_in_data)] was simply synced every second. It just looked like a [uint64|uint32|...]. Then there is an advanced service with different plugins, runs in a separat…

Eh well mine was mostly a hack to index SIP packets. Other existing products just shoved every packet into a MySQL table. One column per field, with indexes. You can imagine that does not scale very well. So I ended up doing similar to what you describe.

Fun how designs converge. You'd think there'd be some generic libraries for this kind of stuff. Now there's LevelDB and its SSTable is sort of close but uses Snappy so it can't operate on compressed data.

What's the common library that implements fast, compressed, immutable key values without lots of per record overhead?

Re: Thoughts on Time-series Databases

#123

Post by pauldix » Wed, Jul 01, 2015, 5:14 PM UTC

mjibson wrote: post #121

pauldix wrote: post #119

Earlier quoted context omitted.

InfluxDB CEO here. Those problems with corrupting data were with the 0.8 line of releases. But to be honest there are people that have been running that and 0.7 in production for almost a year without problems. Your mileage may vary, but we're not supporting any releases prior to the 0.9 line. For the 0.9 set of releases, this is what we're supporting going forward. There are some queries that cause the server to cra…

I performed some OpenTSDB vs InfluxDB comparisons and found that InfluxDB used almost 20x the storage space and was 3x slower than OpenTSDB for an identical data set. The speed isn't that big of a problem and I'm convinced will get faster (esp with the write improvements in 0.9.1), but the space issue is harder to swallow. HBase, which is what backs OpenTSDB, can compress data using LZO or snappy. Uncompressed, our t…

Yep, compression work is starting in the 0.9.2 release cycle. We'll be testing out those compression methods along with other stuff like delta encoding

Re: Thoughts on Time-series Databases

#124

Post by happymcplaksin » Wed, Jul 01, 2015, 11:35 PM UTC

KyleBrandt wrote: post #116

At Stack Exchange our monitoring system bosun ( http://bosun.org ) can use different time series databases as long as they can be bent into tag key+tag value models. Currently it works best with OpenTSDB, but can also support graphite (and elasticsearch populated by logstash). InfluxDB query support is in a branch, but don't want to merge until we have a devoted Bosun+InfluxDB maintainer since we don't use it at Stac…

Have you tried http://prometheus.io/ ?

Re: Thoughts on Time-series Databases

#125

Post by luckydata » Wed, Jul 01, 2015, 11:50 PM UTC

olviko wrote: post #3

What are the benefits of introducing specialized time-series databases vs using Redis, Casandra, or some Sql database?

The answer to this question would be extremely long, so I suggest you first brush up on some fundamentals of database design.

http://www.slideshare.net/bijugs/row-or-columnar-based-datab...

then go onto getting mad at Michael Stonebraker here:

https://gigaom.com/2013/12/14/5-predictions-on-the-future-of...

and then when you realize he's right about everything, you'll have the foundational knowledge to understand the problem a little bit better.

Re: Thoughts on Time-series Databases

#126

Post by engi_nerd » Thu, Jul 02, 2015, 1:44 AM UTC

elektronaut wrote: post #109

engi_nerd wrote: post #93

Earlier quoted context omitted.

Your last sentence is well taken, but the rest leaves me a bit puzzled about what you meant. Your run of the mill laptop is a wonderful machine capable of amazing things. But that laptop is not rated to withstand extreme vibration and shock and huge temperature ranges. Nor does it have the external interfaces to deal with large numbers of custom, high speed digital data buses, nor does it contain the signal condition…

I apologize if it came across as a snarky comment or one-upmanship, I merely intended to support your original comment.

No problem.

Like I said, you do have a very good point. I am often frustrated by the realization that my laptop is much more powerful than this specialized data collection equipment. But we have a very large fixed base of equipment, with no plans to upgrade anytime soon.

Re: Thoughts on Time-series Databases

#127

Post by emmanueloga_ » Thu, Jul 02, 2015, 3:15 AM UTC

I thought someone was going to mention http://druid.io/. I end up not using it for anything, but at some point I was investigating time series DBs and I thought it was interesting.

If I'm not wrong it combines both very fast in-memory OLAP features and what they call "Deep Storage", which I think is a way to store things on disk for slower historical analysis.

Re: Thoughts on Time-series Databases

#128

Post by deathflute » Thu, Jul 02, 2015, 5:27 PM UTC

jimmcslim wrote: post #75

jnordwick wrote: post #71

Earlier quoted context omitted.

I know the Kx people pretty well, and they are trying to get the word out (have been for years), but it never ceases to amaze how little respect they get in the free software world. They are the leading timeseries database, and yet they don't even get a footnote in the article :(

Possibly because they cost an arm and a leg (or at least that's the perception) and are therefore out of reach of most firms, apart from large utilities and hedge funds, and the language looks like line noise. Yes, I know there is a free version, but limited to 32-bit only (and probably non-commercial?). EDIT: 32-bit version can be used commercially.

Agree about the cost. However, I would think that a wider adoption would eventually bring the cost down and perhaps even spawn a bunch of related open-source projects.

As far as readability is concerned, q(KDB+) is far more readable than k(KDB). Also, nobody stops you from adopting a coding style that is more readable. That is what I personally do.

Re: Thoughts on Time-series Databases

#129

Post by pierluca » Thu, Jul 02, 2015, 10:42 PM UTC

robochat wrote: post #42

tsdbase wrote: post #25

Earlier quoted context omitted.

I've hit the same problem and I would like to move back to a SQL data store. However none of the nice dashboards / visualizations support postgres or any SQL database (for now)... My question (to everyone): what do you use as replacement for kibana or grafana?

I've just implemented a custom backend for graphite-api which seems to be working ok although I don't have crazy requirements. https://github.com/brutasse/graphite-api is a cleaned up fork of graphite (which is much easier to install). I'm using grafana as the front-end and my data is in a postgresql database and graphite-api is linking them together.

Hello, I find myself having the same need. Would you agree to share your implementation or point me to it? Thank you!

Re: Thoughts on Time-series Databases

#130

Post by sciurus » Thu, Jul 02, 2015, 11:53 PM UTC

johngd wrote: post #81

sciurus wrote: post #14

If you are looking for time-series databases based on Cassandra that you can use with Graphite, check out Cyanite: https://github.com/pyr/cyanite and https://github.com/brutasse/graphite-cyanite KairosDB: https://github.com/kairosdb/kairosdb and https://github.com/kairosdb/kairos-carbon and https://github.com/Lastik/KairosdbGraphiteFinder

Have you used cyanite in any meaningful way? The original author's (pyr) repo has been pretty dead. This person has been doing a lot of good work: https://github.com/mwmanley/cyanite

I've considered it but haven't made the jump yet. Pyr gave a presentation a few weeks ago that suggested his company is already using it and further development is coming.

https://vimeo.com/131581325