I have looked at InfluxDB, and it is pretty cool. Combined with Grafana, you have a complete solution out of the box.
Ask HN: What DB to use for huge time series?
71–80 of 135 posts
Re: Ask HN: What DB to use for huge time series?
#72Depending on what you are doing you can even try to write it yourself. It would be a good exercise. Here is a toy hand crafted time series storage design: Say you are storing tuples of { , }. Then querying it by timestamp. Writer can store it in two files,open in append only mode only. One is the data file one is the index file. Data might look like: ... And an index file, it stores timestamps and offsets into the da…
I wouldn't recommend trying to do this yourself. Of course you can make something that kind of works, but making a resilient production ready database that is fault tolerant and scales is a lot harder than writing to a file.
Re: Ask HN: What DB to use for huge time series?
#73Use https://crate.io
It is built on Elastic Search and I've recently built something large to store time series data with it. We actually migrated away from Cassandra and ported out application from it because it didn't allow us any schema or indexing flexibility. It also allows you to partition a table by a column (e.g. a day) which means that a new table is created each day. Zero config, fast and operationally easy. Depending on your latency requirements for reads I would also have a serious look at Couchbase but I don't know how well they fare for time-series data.
Re: Ask HN: What DB to use for huge time series?
#74Earlier quoted context omitted.
36-100MB/person per day ~250 days/year expecting ~20,000 (an educated stupid wild ass guess) initially when the system is actually put into production. ~100-400TB per year(?). Most of the data would only be of interest for a month or so, but we do want to preserve the data in general in some usable fashion for testing and some research stuff.
In this case, I would still recommend Cassandra. It can easily handler the data sizes you mention as well as the write rates you imply further down the thread. Cassandra has a nice and simple architecture (every node is identical, no zookeeper roles etc), high write performance and scalability [1], and is fairly robust. My main piece of advice is to get the tables correctly set up. You need to know exactly what queri…
Thanks a ton. I am leaning towards a solution that involves Cassandra. What would you say about using something on top of it like Blueflood?
Re: Ask HN: What DB to use for huge time series?
#75If you're cool with a simple K/V storage format (K: timestamp, V: data), Riak might work for you. Easy to scale, reliable, and wicked fast.
Re: Ask HN: What DB to use for huge time series?
#76First, ask if you really need "massive" scale. Is this an idea, or a well-defined product? I'd imagine if you knew what you were building, you wouldn't be here asking.
So "massive" -- why not prototype on Postgres, and then migrate when you actually have projections on size.
Different orders of magnitude change the technology you work with. Additionally, the latency with which you need to access the metrics (real time, report based).
Cassandra is a pretty solid choice, Influx is really new to the game but is promising.
Druid is trusted by a lot of people, Metamarkets (the author) among them, but may or may not be what you need.
I'd spend some time talking to the people in #druid-dev on Freenode, they're friendly and can help guide you.
Re: Ask HN: What DB to use for huge time series?
#77Not a database but HDF5 ( http://www.hdfgroup.org ) is used for storing all sorts of scientific data, has been around for a while and is very stable. PyTables is built on top of it and there are lots of other languages that can have existing libraries to read/write HDF5 (matlab, python, c, c++, R, java, ...)
I have had good experience using HDF5 to store time series data, but just research datasets and nothing that has been put into production. I don't really know how well it works with threading, for example. It does work very well with PyTables and Pandas for analysis and definitely beats CSV files, which is the the normal way these research datasets are stored.
If you are interested in using HDF5 and PyTables to store time series data, check out this little library that I created: http://andyfiedler.com/projects/tstables-store-high-frequenc...
Re: Ask HN: What DB to use for huge time series?
#78Graphite is a mature system. It's a pain in the ass, but I generally find it essential for server monitoring. I'm working on a timeseries database aimed at replacing graphite. It's just getting started, so it probably won't work immediately, but contributions are welcome. Currently the write performance is already better than graphite [1]. https://github.com/stucchio/timeserieszen [1] This was one of the design goals…
Cool project!
I commented on some other graphite replacement projects at https://news.ycombinator.com/item?id=8368689
Re: Ask HN: What DB to use for huge time series?
#79Earlier quoted context omitted.
Look at the source. (c) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c/k.h (c#) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c.cs This guy is truly depraved.
Dear lord. It must take forever to get up to speed on a codebase like that.
Doesn't take that long. Maybe a few days?
k5 isn't that big (about 9 C files)
Re: Ask HN: What DB to use for huge time series?
#80kdb+/q is commonly used in the financial world for these types of problems. They have a 32bit for free, and you can ask them about pricing on the 64bit version. http://kx.com/
Is this just your typical columnar OLAP database, like Vertica or Big Query?