Redshift or Vertica are your best options and are built for massive queries over large data.
Ask HN: What DB to use for huge time series?
121–130 of 135 posts
Re: Ask HN: What DB to use for huge time series?
#122Good talk at Strangeloop this year about exactly this. https://thestrangeloop.com/sessions/time-series-data-with-ap...
The talk is posted here. https://www.youtube.com/watch?v=ovMo5pIMj8M
Re: Ask HN: What DB to use for huge time series?
#123Chiming in with a definite bias. I'm one of the co-founders of InfluxDB, and while we're still somewhat young, we actually just hit the 1-year anniversary of our first commit today. We're currently a team of 5 full-time developers, dedicated to making InfluxDB the best time series database available. We've also got some strong institutional backing, so we're not going anywhere for a very, very long time.
If there are any questions we can answer to help you make a more informed decision, drop us a line at support@influxdb.com or reach out to the community: https://groups.google.com/d/forum/influxdb
Re: Ask HN: What DB to use for huge time series?
#124Use the ELK Stack - Elasticsearch, Logstash, Kibana. Logstash is for ETL and data normalization. Kibana is for building cool visualizations. Elasticsearch for storing, processing, analysis, scaling and search.
Here are some resources:
Webinar: the Elk Stack in a Devops Environment http://www.elasticsearch.org/webinars/elk-stack-devops-envir...
Webinar: An Introduction to the ELK Stack http://www.elasticsearch.org/webinars/introduction-elk-stack...
Re: Ask HN: What DB to use for huge time series?
#125I'd recommend OpenTSDB. Using a 11 node hadoop cluster on m1.xlarge nodes in Amazon, (2 name, 9 data), I can ingest a sustained rate of ~75,000 time series datapoints per second in an HBase table. The upside is that OpenTSDB scales really well with hadoop cluster size, so you can just scale it up to handle more load. The downsides are that their data schema and query format are optimized for data efficiency, not spee…
this, or Hbase + Phoenix: http://phoenix.apache.org/
Re: Ask HN: What DB to use for huge time series?
#126Depending on what you are doing you can even try to write it yourself. It would be a good exercise. Here is a toy hand crafted time series storage design: Say you are storing tuples of { , }. Then querying it by timestamp. Writer can store it in two files,open in append only mode only. One is the data file one is the index file. Data might look like: ... And an index file, it stores timestamps and offsets into the da…
I wouldn't recommend trying to do this yourself. Of course you can make something that kind of works, but making a resilient production ready database that is fault tolerant and scales is a lot harder than writing to a file.
Well it was just a toy example I came up with in a couple of minutes.
But sometimes depending on the requirements a file is enough. If you intimately know the and control the bytes that get written it is easier to understand and reason about your systems (that means optimizing it, scaling it, making it fault tolerant).
Also one way to make a resilient and fault tolerant database is to have less code running. Sometimes the base libc and unix offer a good and stable base on which it is easy to build. If you append the file in read or append only mode. You can rely on certain behavior now.
People in the past have bought into marketing crap and got stuff like MongoDB which would throw data over the fence and pray that it would be synced eventually (by default!). But heck it was WebScale(tm).
Re: Ask HN: What DB to use for huge time series?
#127Earlier quoted context omitted.
I wouldn't recommend trying to do this yourself. Of course you can make something that kind of works, but making a resilient production ready database that is fault tolerant and scales is a lot harder than writing to a file.
Well it was just a toy example I came up with in a couple of minutes. But sometimes depending on the requirements a file is enough. If you intimately know the and control the bytes that get written it is easier to understand and reason about your systems (that means optimizing it, scaling it, making it fault tolerant). Also one way to make a resilient and fault tolerant database is to have less code running. Sometime…
That is why you seriously audit your tools and why many in the industry avoid Mongo like the plague. Controlling the byes that gets written to a file is actually not simple at all, and its a huge research problem as far as file systems of databases. I'm just saying, I don't think writing your own db is every a very good idea, unless it is SO simple that you would barely call it a DB.
Re: Ask HN: What DB to use for huge time series?
#128Cassandra works best. E.g. Say as the backend for email storage.
Re: Ask HN: What DB to use for huge time series?
#129Depending on how 'huge' your timeseries are, you might be pleasantly surprised with Postgres. Postgres scales to multiple TB just fine, and of course the software can be easier to write since you have SQL and ORMs to rely on. It's also an incredibly mature and stable software package, if you're worried about future-proofing. Some (constantly-growing) timeseries can be stored on a per-row basis, while other (static or…
>I find that most of the time, "Big Data" isn't really all that big for modern hardware, and so going through all of the extra software work for specialized data stores isn't really all that necessary. YMMV, of course, depending on the nature of your queries.
I totally agree. Most of useful "big data" is time-series data, and they aren't all that huge compared to images/videos/etc.
That being said, I think the reason to adopt something like Hadoop/MPP engines is not for storage but ease of querying: while Postgres can handle storing terabytes of data, joining two terabyte-scale tables can get a little iffy. This gets even more complex if you start packing data into array columns for space efficiency.
There is an argument to be made that historical/archival data aren't all that useful and thus do not need to be analyzed: that was definitely my assumption coming from finance. However, I've been surprised how far back some of our customers at Treasure Data go to mine insights from data.
Re: Ask HN: What DB to use for huge time series?
#130My company is currently using Mongo, and while it works, I wouldn't recommend it. We're looking at Cassandra and Elasticsearch, which seems to be a lot more promising.
The number of horror stories I've seen about mongo is up to around 10 this month alone. I'm now glad I never made the jump... in the meantime, pgsql is still on my list
We had a 50gb instance initially and it was no problem, then our app started to get a lot more traffic in a short period of time. We had to start sharding on a reasonably large scale. Mongo is a lot harder to maintain when it's big and unpredictable. I know this sounds like a plug, alas it's the truth, we found Object Rocket and now we don't worry about Mongo.