91–100 of 135 posts
I'd recommend OpenTSDB. Using a 11 node hadoop cluster on m1.xlarge nodes in Amazon, (2 name, 9 data), I can ingest a sustained rate of ~75,000 time series datapoints per second in an HBase table. The upside is that OpenTSDB scales really well with hadoop cluster size, so you can just scale it up to handle more load. The downsides are that their data schema and query format are optimized for data efficiency, not spee…
KDB+ http://kx.com/kdb-plus.php I have no affiliation, other than being a customer. Its as close to a standard as you can find in finance. There are many useful tutorials out there that let you try it out and you can usually get an eval version to try before you buy. http://code.kx.com/wiki/Startingkdbplus/contents If you find something that is comparable in terms of performance and features, but cheaper, please mail…
If in doubt, start with a traditional RDBMS. And ONLY after you profile your application and see exactly where your pain points are, do something crazy.
Have fun!
Earlier quoted context omitted.
When it comes to time series, reasoning in terms of byte size does not really make sense, it's better to state how many datapoints you need to handle and in how many distinct time series they are distributed.
8-16ish datapoints per sample and they'll be distributed more or less evenly during the day and then pretty much go dead at night. There may or may not be a value for every data point at every sample.
Bad news is picking a system means understanding access patterns -- reading, not writing. Do you only need to look within a single user? That's much easier. If you have to query across users, or do things like (and I have no idea what your problem domain is, but if it's utility usage, things like average usage by zip or block; if it's wearables, activity by city, etc), stuff gets much harder. How granular do you need to be able to query, and how far back? What is the sla on a query: are results calculated in batch mode or on demand for a website? You often have to duplicate data in order to optimize one set for throughput access and the other set for minimal random query time. Can you get away with logarithmic granularity for queries, ie every sample is available for 1 month, every 3rd for the next month, every 10th for a couple months after that, etc. What windowing functions do you need to run, and how frequently do they need to be updated? What is the ratio of writes to reads? If you have to access random data quickly, eg for a site, can you calculate > 1 day back in batch mode, cache those results, and add the last 24h of data at runtime? etc etc etc.
You need to have some conversations with the data consumers.
Edit: and I've assumed these data are read-only; if you can update them, then there's far more difficulty.
I have a timeseries problem on the backburner, and like you am hopeful for InfluxDB but it's still missing a couple features that I need, so haven't used it yet. As another person mentioned, you're going to be looking at columnar databases (few/one rows, with a very large amount of columns) if you have truly large storage requirements. Since my data is still small, I'm sticking with Postgres for now. I've seen a coup…
What features are you from InfluxDB? I am a long-time graphite user, and I just saw InfluxDB, and it looked really good.
[1]: https://github.com/influxdb/influxdb/issues/68
[2]: http://influxdb.com/docs/v0.8/api/continuous_queries.html
Anyway, no one has mentioned RRD tool yet: http://oss.oetiker.ch/rrdtool/
"RRDtool is the OpenSource industry standard, high performance data logging and graphing system for time series data. RRDtool can be easily integrated in shell scripts, perl, python, ruby, lua or tcl applications."
Earlier quoted context omitted.
8-16ish datapoints per sample and they'll be distributed more or less evenly during the day and then pretty much go dead at night. There may or may not be a value for every data point at every sample.
There's good news and bad news. Good news is storing this much data isn't hard; there's plenty of people who've done it and many systems will scale enough. Bad news is picking a system means understanding access patterns -- reading, not writing. Do you only need to look within a single user? That's much easier. If you have to query across users, or do things like (and I have no idea what your problem domain is, but i…
So many people suggesting relational databases or just plain "big data" solutions. Time series databases tend to have quite unique features like interpolation of data (i.e. you can query a specific datapoint at a specific date and time for a value, and you will get an interpolated value if there is no specific sample for that data point.) Anyway, no one has mentioned RRD tool yet: http://oss.oetiker.ch/rrdtool/ "RRDt…