Redshift. It scales superbly.
Ask HN: What DB to use for huge time series?
21–30 of 135 posts
Re: Ask HN: What DB to use for huge time series?
#22kdb+/q is commonly used in the financial world for these types of problems. They have a 32bit for free, and you can ask them about pricing on the 64bit version. http://kx.com/
Yep, banks heavily use these to store tickers (currencies, instruments) and do calculations of VWAP, etc. Pretty much standard in the industry for these kind of applications.
Re: Ask HN: What DB to use for huge time series?
#23If this is for a Manufacturing environment, OSI Soft's PI Historian may be a good fit. Really depends on what exactly your requirements are.
Re: Ask HN: What DB to use for huge time series?
#24My company is currently using Mongo, and while it works, I wouldn't recommend it. We're looking at Cassandra and Elasticsearch, which seems to be a lot more promising.
Beware of using Elasticsearch as a primary DB. Kyle Kingsbury has shown that it loses an awful lot of data during a partition, despite their claims. Example: http://aphyr.com/posts/317-call-me-maybe-elasticsearch
Re: Ask HN: What DB to use for huge time series?
#25Depending on how 'huge' your timeseries are, you might be pleasantly surprised with Postgres. Postgres scales to multiple TB just fine, and of course the software can be easier to write since you have SQL and ORMs to rely on. It's also an incredibly mature and stable software package, if you're worried about future-proofing. Some (constantly-growing) timeseries can be stored on a per-row basis, while other (static or…
Some tips for querying timeseries in Postgres: http://no0p.github.io/postgresql/2014/05/08/timeseries-tips-...
HN discussion: https://news.ycombinator.com/item?id=7809819
Re: Ask HN: What DB to use for huge time series?
#26Until recently, TempoIQ used Apache HBase to store time series data. http://blog.tempoiq.com/why-tempoiq-moved-off-hbase
Re: Ask HN: What DB to use for huge time series?
#27For a good answer, you need to provide a lot more detail in the requirements:
- What do the writes look like? If they are coming in a stream how many writes per second do you need to support? If they are a bulk load how large and frequent are the batches? Simple numerical values?
- What do the reads look like? How many queries per second do you need to support? How much data per query? How fast do the queries need to be? Will your queries be simple aggregations? Dimensional queries? Unique dimension value counts? Are approximations tolerated?
- How much history do you need to keep?
- What are your requirements for availability?
- What are your requirements for consistency?
- How fast does new data have to show up in reads?
Without more detail, you're going to get dozens of suggestions which may each be right for a particular case.
Re: Ask HN: What DB to use for huge time series?
#28My company is currently using Mongo, and while it works, I wouldn't recommend it. We're looking at Cassandra and Elasticsearch, which seems to be a lot more promising.
I've only had about a year and a half of working in that space, but can confirm that Cassandra's quite good for truly huge time-series data. If you want to record an event every time someone makes a purchase at a Wal-Mart, turn to Cassandra or a similar system.
It is also much more highly regarded as a primary data store than Elasticsearch.
Re: Ask HN: What DB to use for huge time series?
#29Maybe check this out?
https://github.com/soundcloud/roshi
Roshi is basically a high-performance index for timestamped data. It's designed to sit in the critical (request) path of your application or service. The originating use case is the SoundCloud stream; see this blog post for details.
Re: Ask HN: What DB to use for huge time series?
#30I have a timeseries problem on the backburner, and like you am hopeful for InfluxDB but it's still missing a couple features that I need, so haven't used it yet.
As another person mentioned, you're going to be looking at columnar databases (few/one rows, with a very large amount of columns) if you have truly large storage requirements. Since my data is still small, I'm sticking with Postgres for now.
I've seen a couple people mention OpenTSDB; another alternative to that is KairosDB[1], which adds Cassandra support and focuses on data purity[2] (OpenTSDB will interpolate values if there are holes).
And to echo another person, just forget about Graphite/Whisper. It uses a simple pre-allocated block format that will eventually cause problems when you want to change time windows.