Not a database but HDF5 (http://www.hdfgroup.org) is used for storing all sorts of scientific data, has been around for a while and is very stable. PyTables is built on top of it and there are lots of other languages that can have existing libraries to read/write HDF5 (matlab, python, c, c++, R, java, ...)
Ask HN: What DB to use for huge time series?
31–40 of 135 posts
Re: Ask HN: What DB to use for huge time series?
#32There's a company in Chicago called TempoIQ (formerly TempoDB) that is working on a time series database.
I'm not affiliated with them, I just met them once.
Re: Ask HN: What DB to use for huge time series?
#33First question - do you even need a database right now? Have you for example considered using CSV files and simply loading those files into Pandas or R on demand? I am currently working on a project analyzing massive amounts of options data and have found this approach to be both quite easy as well as flexible to work with... and as my project matures I may move select parts of it into a database.
>loading those files into Pandas or R
What is "massive" for you? I was under impression you can't use R or pandas for anything that doesn't fit into memory.
Re: Ask HN: What DB to use for huge time series?
#34It would be useful to know what "huge" means here. And how you want to look up the data. That said, I've used Cassandra in the past for timeseries data as one of the useful queries that can be made is a range query (if the composite key is set up correctly)
36-100MB/person per day ~250 days/year expecting ~20,000 (an educated stupid wild ass guess) initially when the system is actually put into production. ~100-400TB per year(?). Most of the data would only be of interest for a month or so, but we do want to preserve the data in general in some usable fashion for testing and some research stuff.
Re: Ask HN: What DB to use for huge time series?
#35Check out KairosDB... it is based on Cassandra and is very similar to OpenTSDB but IMO Cassandra is a bit easier to scale and maintain with fewer parts.
We're using it in production... it's still early but there are about 1-2 dozen moderate sized installs (like 10 box installs).
We're pretty happy with it so far..
Re: Ask HN: What DB to use for huge time series?
#36For a good answer, you need to provide a lot more detail in the requirements: - What do the writes look like? If they are coming in a stream how many writes per second do you need to support? If they are a bulk load how large and frequent are the batches? Simple numerical values? - What do the reads look like? How many queries per second do you need to support? How much data per query? How fast do the queries need to…
All these questions should ideally not be a concern when you are looking for a database. A general purpose database which can handle all the above and more is AmisaDB. http://www.amisalabs.com/
Re: Ask HN: What DB to use for huge time series?
#37Blueflood(http://blueflood.io/) may be what you are looking for. It uses Cassandra under the hood. It's a project out of Rackspace and is being used in prod by Rackspace's cloud monitoring. Currently, Blueflood ingests about 2.2M metrics/min and can probably scale to 40M metrics/min. Full disclosure - I am a dev on that project. It's being actively developed!
Re: Ask HN: What DB to use for huge time series?
#38Blueflood(http://blueflood.io/) may be what you are looking for. It uses Cassandra under the hood. It's a project out of Rackspace and is being used in prod by Rackspace's cloud monitoring. Currently, Blueflood ingests about 2.2M metrics/min and can probably scale to 40M metrics/min. Full disclosure - I am a dev on that project. It's being actively developed!
Re: Ask HN: What DB to use for huge time series?
#39Postgres is fantastic for most things. I think people think "big data" is somehow too big, the big data I've seen fits in a few terabytes which postgres handles just fine
Re: Ask HN: What DB to use for huge time series?
#40For a good answer, you need to provide a lot more detail in the requirements: - What do the writes look like? If they are coming in a stream how many writes per second do you need to support? If they are a bulk load how large and frequent are the batches? Simple numerical values? - What do the reads look like? How many queries per second do you need to support? How much data per query? How fast do the queries need to…
Part of the reason the question was light on details is that this is just at the very beginning and a lot of relevant things aren't locked in yet. Below are the back of napkin results and are subject to the risk of being laughably wrong.
Writes: not totally sure in terms of how the data is being packaged before being sent yet, but it'll probably be more than 10 writes a second but less than 1000 initially(?). Not sure yet if we're aggregating and batching before sending or if we are, to what degree.
Availability: If it has brief breaks where it just misses some data (Reads will likely be grabbing the last n records of a given set of sensors maybe with some light math on it if the query language supports it, though there might be an easier way to cache recent history and then only need to go to the big list for responding to a longer-term issue. Also the nature of reads is very subject to change since there's a bunch of use-cases for the data being kicked around and I haven't gone through what each use's reads would look like yet.
New data needs to show up in reads in soft-real time. The napkin-estimate indicates that we might be looking at asking for about 6-80MB returned per query as a generally large but perhaps not max query, bigger operations that dealt with legitimately huge amounts of data will probably be scheduled around lighter periods/put on different machines (not sure how adding more machines reading would impact since I don't know what db it will be yet).
Ideally keep as much history as humanly possible, possibly moving them to physical archival at some point (1yr+?).