have a look at https://dalmatiner.io/ I think it's used here https://project-fifo.net/
Ask HN: What DB to use for huge time series?
61–70 of 135 posts
Re: Ask HN: What DB to use for huge time series?
#62I'd recommend OpenTSDB. Using a 11 node hadoop cluster on m1.xlarge nodes in Amazon, (2 name, 9 data), I can ingest a sustained rate of ~75,000 time series datapoints per second in an HBase table.
The upside is that OpenTSDB scales really well with hadoop cluster size, so you can just scale it up to handle more load.
The downsides are that their data schema and query format are optimized for data efficiency, not speed or flexibility. It's really easy to refine a search for a particular metric by filtering on tags, but it's really hard to do any sort of analysis across metrics, so you have to write your own glue on top of that which fetches the datapoints for the metrics you care about, and does its own aggregation.
Re: Ask HN: What DB to use for huge time series?
#63Blueflood( http://blueflood.io/ ) may be what you are looking for. It uses Cassandra under the hood. It's a project out of Rackspace and is being used in prod by Rackspace's cloud monitoring. Currently, Blueflood ingests about 2.2M metrics/min and can probably scale to 40M metrics/min. Full disclosure - I am a dev on that project. It's being actively developed!
If you are considering software-as-a-service solution, Rackspace has just released public APIs of Cloud Metrics powered by blueflood at no additional cost. http://www.rackspace.com/blog/cloud-metrics-working-toward-a... (Disclaimer: I am the Product Manager on that project)
This is a fantastic snapshot of an engineer and PM commenting on the same product
Re: Ask HN: What DB to use for huge time series?
#64KDB+ http://kx.com/kdb-plus.php I have no affiliation, other than being a customer. Its as close to a standard as you can find in finance. There are many useful tutorials out there that let you try it out and you can usually get an eval version to try before you buy. http://code.kx.com/wiki/Startingkdbplus/contents If you find something that is comparable in terms of performance and features, but cheaper, please mail…
Look at the source. (c) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c/k.h (c#) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c.cs This guy is truly depraved.
My favorite part is the comment "remove more clutter." Instead it should have been, "Remove all hope of maintenance."
Re: Ask HN: What DB to use for huge time series?
#65Have a look at TempoDB - built specifically for timeseries data ( https://tempo-db.com/about/ )
TempoDB has renamed itself into TempoIQ and no longer offer their storage service. I've heard some angry comments from customers who recently received an email telling them the storage service they were using was to be shutdown at the end of october!
I work at TempoIQ, and we still to offer our storage service. We've launched a new product (as TempoIQ) that is hosted in a private environment and offers storage, historical analysis, and real-time monitoring.
As for the customers on TempoDB, we are working them to transition to TempoIQ if the switch makes sense or offering to guide them in a transition to another time-series database like InfluxDB.
Re: Ask HN: What DB to use for huge time series?
#66Graphite is a mature system. It's a pain in the ass, but I generally find it essential for server monitoring.
I'm working on a timeseries database aimed at replacing graphite. It's just getting started, so it probably won't work immediately, but contributions are welcome. Currently the write performance is already better than graphite [1].
https://github.com/stucchio/timeserieszen
[1] This was one of the design goals. Whenever graphite receives a data point a disk seek is incurred - the data point must be appended to the timeseries file. Timeserieszen uses a WAL - data flowing in is immediately written, and periodically the WAL is rolled over into permanent storage.
Re: Ask HN: What DB to use for huge time series?
#67Depending on what you are doing you can even try to write it yourself. It would be a good exercise.
Here is a toy hand crafted time series storage design:
Say you are storing tuples of {,}. Then querying it by timestamp.
Writer can store it in two files,open in append only mode only. One is the data file one is the index file. Data might look like:
...
And an index file, it stores timestamps and offsets into the data files where the blobs are:
...
If you need rolling fall-off. Then create new pairs of files every day (hour, week, month). And delete old ones as you go.
Then if you can ensure that your have time synchronization set up and timestamp are in increasing order (this might be hard). You can do binary searching. If you use rolling fall-offs. Then you can discard whole files periods based on the query range when you search.
All this would go into a directory. Reader and writer could be different processes. Your timestamp and offset sizes should be fixed length. Writer first appends to the data file and then writes the index. Reader knows how to find the last valid record by looking at the size of the file.
Re: Ask HN: What DB to use for huge time series?
#68My company is currently using Mongo, and while it works, I wouldn't recommend it. We're looking at Cassandra and Elasticsearch, which seems to be a lot more promising.
The number of horror stories I've seen about mongo is up to around 10 this month alone. I'm now glad I never made the jump... in the meantime, pgsql is still on my list
I wouldn't say it's a horror story, it's just not really for time series "big data". The backend guys have had to muck about with the data a lot to get good performance out of it. There's some optimizations we missed on the sysadmin side too, like sharding the cluster after it got to ~250GB, and now it's many times that. Our Mongo clusters have been running production for well over a year.
Re: Ask HN: What DB to use for huge time series?
#69It would be useful to know what "huge" means here. And how you want to look up the data. That said, I've used Cassandra in the past for timeseries data as one of the useful queries that can be made is a range query (if the composite key is set up correctly)
36-100MB/person per day ~250 days/year expecting ~20,000 (an educated stupid wild ass guess) initially when the system is actually put into production. ~100-400TB per year(?). Most of the data would only be of interest for a month or so, but we do want to preserve the data in general in some usable fashion for testing and some research stuff.
In this case, I would still recommend Cassandra. It can easily handler the data sizes you mention as well as the write rates you imply further down the thread.
Cassandra has a nice and simple architecture (every node is identical, no zookeeper roles etc), high write performance and scalability [1], and is fairly robust. My main piece of advice is to get the tables correctly set up. You need to know exactly what queries you want to make and design a table around that query (Cassandra only allows performant queries to be made, unless you go out of your way to set a flag). Whether a query is possible or performant depends on the key of the rows for the table, which may be a composite key. Take a look at the cassandra documentation for more details.
1. http://techblog.netflix.com/2011/11/benchmarking-cassandra-s...
Re: Ask HN: What DB to use for huge time series?
#70It would be useful to know what "huge" means here. And how you want to look up the data. That said, I've used Cassandra in the past for timeseries data as one of the useful queries that can be made is a range query (if the composite key is set up correctly)
36-100MB/person per day ~250 days/year expecting ~20,000 (an educated stupid wild ass guess) initially when the system is actually put into production. ~100-400TB per year(?). Most of the data would only be of interest for a month or so, but we do want to preserve the data in general in some usable fashion for testing and some research stuff.
You might look into partitioning. Oracle and SQL Server both support that type of operation. Additionally, being able to find support when things get "too big to handle" can be easier on a mature technology with lots of users.
On a side note, you can hook a Hadoop cluster up to SQL Server if you're into that kind of thing for storage.