I think Elasticsearch is a good solution for analytic workloads (including time-series data). Query speeds are significantly faster than most DBs because it uses the vector space model (which also introduces the possibility of false positives). I wouldn't recommend it (yet) as a primary data store. It is however really useful for analytics.
Ask HN: What DB to use for huge time series?
51–60 of 135 posts
Re: Ask HN: What DB to use for huge time series?
#52What kind of project is that? Is it a side project or one that will give birth to a company? Requirements are rather different depending on importance you give to your (or your customers') data.
It's for work. If it were a side project I would probably have just grabbed InfluxDB and run with that since it looks the most fun, but since it's for a core part of the whole system then the risk of project abandonment is a bit high.
Re: Ask HN: What DB to use for huge time series?
#53To get a relevant recommendation, you'll have to describe two things at least - data and queries. How is the data generated? What is stored in the data? What are the queries you plan to run?
The data is going to be sensor readings. Just numeric readouts over time. There will probably be about 8-16 physical sensor points per person per reading. I'll want to retrieve slices of time rather than individual records and likely produce some averages/basic algebra over those slices in order to produce more meaningful data for the rest of the system which is pretty vanilla in terms of data requirements.
I made a data star supporting weather/water sensor data in postgresql that heavily relied on table partitions for handling performance [1]. We had it on pretty weak machines, replicated with bucardo, and never had any issues. It worked well to several million records/month (not sure where it is now).
[1] https://github.com/imperialwicket/postgresql-time-series-tab...
Re: Ask HN: What DB to use for huge time series?
#54If your problem is storing lots of metric points over time, I'd like to toss in something here: https://dalmatiner.io . It has an effective size of about 9 bits (bit not byte) per time interval (i.e. per second). Adding data and querying data are both very fast (as in a 'give me the daily average of cpu usage on that system for the last week' takes about 2ms) and it is built to be distributed from the ground up (using riak_core the same mechanism used by the riak database).
Some benchmarks to get an idea: https://docs.dalmatiner.io/en/latest/benchmarks.html
disclaimer: I’m the author of DalmatinerDB
Re: Ask HN: What DB to use for huge time series?
#55KDB+ http://kx.com/kdb-plus.php I have no affiliation, other than being a customer. Its as close to a standard as you can find in finance. There are many useful tutorials out there that let you try it out and you can usually get an eval version to try before you buy. http://code.kx.com/wiki/Startingkdbplus/contents If you find something that is comparable in terms of performance and features, but cheaper, please mail…
Look at the source.
(c) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c/k.h (c#) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c.cs
This guy is truly depraved.
Re: Ask HN: What DB to use for huge time series?
#56DataDog uses elasticsearch for their timeseries data store: http://www.elasticsearch.org/content/uploads/2013/11/es_case...
Elasticsearch might seem like a strange option at first since it's historically a text search engine, but it's main datastructure is a compressed bit array which is ideal for OLAP processing.
Re: Ask HN: What DB to use for huge time series?
#57Cassandra was used at Twitter[0] to store quite a lot of time series data.
A typical production instance of the time series database is based on four distinct Cassandra clusters, each responsible for a different dimension (real-time, historical, aggregate, index) due to different performance constraints. These clusters are amongst the largest Cassandra clusters deployed in production today and account for over 500 million individual metric writes per minute. Archival data is stored at a lower resolution for trending and long term analysis, whereas higher resolution data is periodically expired.
Re: Ask HN: What DB to use for huge time series?
#58KDB+ http://kx.com/kdb-plus.php I have no affiliation, other than being a customer. Its as close to a standard as you can find in finance. There are many useful tutorials out there that let you try it out and you can usually get an eval version to try before you buy. http://code.kx.com/wiki/Startingkdbplus/contents If you find something that is comparable in terms of performance and features, but cheaper, please mail…
Look at the source. (c) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c/k.h (c#) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c.cs This guy is truly depraved.
That is special. I had a fun time parsing that. I mean that genuinely.
Re: Ask HN: What DB to use for huge time series?
#59KDB+ http://kx.com/kdb-plus.php I have no affiliation, other than being a customer. Its as close to a standard as you can find in finance. There are many useful tutorials out there that let you try it out and you can usually get an eval version to try before you buy. http://code.kx.com/wiki/Startingkdbplus/contents If you find something that is comparable in terms of performance and features, but cheaper, please mail…
Look at the source. (c) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c/k.h (c#) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c.cs This guy is truly depraved.
Dear lord. It must take forever to get up to speed on a codebase like that.
Re: Ask HN: What DB to use for huge time series?
#60KDB+ http://kx.com/kdb-plus.php I have no affiliation, other than being a customer. Its as close to a standard as you can find in finance. There are many useful tutorials out there that let you try it out and you can usually get an eval version to try before you buy. http://code.kx.com/wiki/Startingkdbplus/contents If you find something that is comparable in terms of performance and features, but cheaper, please mail…
Look at the source. (c) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c/k.h (c#) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c.cs This guy is truly depraved.
Hah... Almost looks like the programmer started out trying to create an entry for obfuscated C competition but then pivoted midway when s/he realized it could be commercialized.
Specially loved this comment:
// remove more clutter
#define O printf
#define R return
#define Z static