Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What DB to use for huge time series?

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 135 posts

Re: Ask HN: What DB to use for huge time series?

#111

Post by pnmahoney »

BWStearns wrote: post #48
hbs wrote: post #45

Earlier quoted context omitted.

When it comes to time series, reasoning in terms of byte size does not really make sense, it's better to state how many datapoints you need to handle and in how many distinct time series they are distributed.

8-16ish datapoints per sample and they'll be distributed more or less evenly during the day and then pretty much go dead at night. There may or may not be a value for every data point at every sample.

My guess would be you would want Cassandra, specifically to incur less overhead for empty values. I haven't built finance backtesting/monitoring infrastructure - which sounds exactly like what you're building - but in this case, I think you'll get real value from triggers, even if that's only being supported experimentally right now.

Re: Ask HN: What DB to use for huge time series?

#112

Post by gourneau »

gourneau wrote: post #20

I have used http://influxdb.com/ , I have used it with a few million records. Getting the data out is a bit slow because it goes over HTTP. Also make your InfluxDB library of choice can deal with HTTP chunking. I found that if you request a lot of data from InfluxDB and the system does not have enough memory, the process will silently die. If you have mega huge data http://opentsdb.net/ seems pretty decent, however I…

Clarification: InfluxDB only crashed on me when I requested a lot of data without chunking. With chunking I didn't have any problems.

I like InfluxDB and still use it.

Re: Ask HN: What DB to use for huge time series?

#113

Post by lobster_johnson »

sciurus wrote: post #18
ankushio wrote: post #5

Have you considered opentsdb or graphite? I love graphite because of the nice frontend interface and functionality it provides for visualizing and transforming your metrics.

Development of graphite is effectively dead. The datastore component (carbon and whisper) has design issues, and the official replacement (ceres) hasn't seen any commits this year. There are a some alternatives, though. For data storage Cyanite [0] speaks the graphite protocol and stores the data in Cassandra. Alternately, InfluxDB [1] speaks the graphite protocol and stores the data in itself To get the data back ou…

A slightly off-topic question, since you seem to know what you're talking about: What are people using these days for collecting and display devops-level metrics, if it's not Graphite? Are your links relevant here?

Last I looked at Graphite I balked at the data store design (very I/O heavy) and the awful front ends (very limited graphing and reporting capabilities). But I haven't discovered a good alternative that has traciton. Diamond seems like the thing to use for collecting metrics (instead of collectd), though.

Edit: Grafana looks good, actually.

Re: Ask HN: What DB to use for huge time series?

#115

Post by capkutay »

spiralganglion wrote: post #24
caw wrote: post #3

My company is currently using Mongo, and while it works, I wouldn't recommend it. We're looking at Cassandra and Elasticsearch, which seems to be a lot more promising.

Beware of using Elasticsearch as a primary DB. Kyle Kingsbury has shown that it loses an awful lot of data during a partition, despite their claims. Example: http://aphyr.com/posts/317-call-me-maybe-elasticsearch

I think with $70m+ in funding, ES has the resources to fix their split brain issues.

Re: Ask HN: What DB to use for huge time series?

#116

Post by TheAceOfHearts »

hbs wrote: post #46
TheAceOfHearts wrote: post #7

If you're up for considering a cloud service, you might want to check out Treasure Data ( http://treasuredata.com/ ). The free plan allows 10M records per month with a maximum capacity of 150M. Full disclosure: I work there.

Does treasure data have a dedicated storage engine for time series? This kind of data has specific needs which are not met by general purpose storage layers.

To an extent, yes. We wrote our time-partitioned columnar storage from scratch: it has row-based storage for more recent data and column-based storage for historical data, and the data is merged from row-based to column-based periodically for performance. We realized from day one that much of "big data" is log/timestamped data, so our query execution engined are optimized for time-windowed queries.

Re: Ask HN: What DB to use for huge time series?

#117

Post by didip »

The big idea in storing time series data is to partition data by timestamp (daily/hourly/minutely depending on how granular do you want it). This technique can be done in various data store:

* I've done it in PostgreSQL using triggers and table inheritance. With this technique trimming old data is as simple as dropping old tables.

* Logstash folks use daily indices on ElasticSearch to store log data which is time series by nature.

* I have heard from quite a few people that Cassandra works really well with this data model too.

Re: Ask HN: What DB to use for huge time series?

#119

Post by scottlocklin »

chollida1 wrote: post #41

KDB+ http://kx.com/kdb-plus.php I have no affiliation, other than being a customer. Its as close to a standard as you can find in finance. There are many useful tutorials out there that let you try it out and you can usually get an eval version to try before you buy. http://code.kx.com/wiki/Startingkdbplus/contents If you find something that is comparable in terms of performance and features, but cheaper, please mail…

Jd (the J database) isn't as powerful as KDB+, but it's pretty good and only costs if you want to pay for support.

Re: Ask HN: What DB to use for huge time series?

#120

Post by scottlocklin »

maslam wrote: post #21

Redshift. It scales superbly.

While KDB+ is infinitely better for TS data, hiring people who know what they're doing, and buying the physical hardware you need to make it fly isn't what most modern firms are interested in. If you want to run on the EC2, Redshift is super great. Time/date range queries though, holy shit those suck.
Post reply on HN