41–50 of 135 posts
I have no affiliation, other than being a customer. Its as close to a standard as you can find in finance.
There are many useful tutorials out there that let you try it out and you can usually get an eval version to try before you buy.
http://code.kx.com/wiki/Startingkdbplus/contents If you find something that is comparable in terms of performance and features, but cheaper, please mail me!! I would be very grateful.
If your format is cast in stone you may also be able to get away with using flat-files. If you implement the List interface or something similar it would be very easy to integrate into your application. (Normally I wouldn't recommend flat-files for anything, but for time series it can be not a bad option, as much as that makes me cringe).
Blueflood( http://blueflood.io/ ) may be what you are looking for. It uses Cassandra under the hood. It's a project out of Rackspace and is being used in prod by Rackspace's cloud monitoring. Currently, Blueflood ingests about 2.2M metrics/min and can probably scale to 40M metrics/min. Full disclosure - I am a dev on that project. It's being actively developed!
http://www.rackspace.com/blog/cloud-metrics-working-toward-a...
(Disclaimer: I am the Product Manager on that project)
It would be useful to know what "huge" means here. And how you want to look up the data. That said, I've used Cassandra in the past for timeseries data as one of the useful queries that can be made is a range query (if the composite key is set up correctly)
36-100MB/person per day ~250 days/year expecting ~20,000 (an educated stupid wild ass guess) initially when the system is actually put into production. ~100-400TB per year(?). Most of the data would only be of interest for a month or so, but we do want to preserve the data in general in some usable fashion for testing and some research stuff.
If you're up for considering a cloud service, you might want to check out Treasure Data ( http://treasuredata.com/ ). The free plan allows 10M records per month with a maximum capacity of 150M. Full disclosure: I work there.
Have a look at TempoDB - built specifically for timeseries data ( https://tempo-db.com/about/ )
Earlier quoted context omitted.
36-100MB/person per day ~250 days/year expecting ~20,000 (an educated stupid wild ass guess) initially when the system is actually put into production. ~100-400TB per year(?). Most of the data would only be of interest for a month or so, but we do want to preserve the data in general in some usable fashion for testing and some research stuff.
When it comes to time series, reasoning in terms of byte size does not really make sense, it's better to state how many datapoints you need to handle and in how many distinct time series they are distributed.
Earlier quoted context omitted.
When it comes to time series, reasoning in terms of byte size does not really make sense, it's better to state how many datapoints you need to handle and in how many distinct time series they are distributed.
8-16ish datapoints per sample and they'll be distributed more or less evenly during the day and then pretty much go dead at night. There may or may not be a value for every data point at every sample.