Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What DB to use for huge time series?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 135 posts

Re: Ask HN: What DB to use for huge time series?

#81

Post by Xorlev »

spudlyo wrote: post #57

Cassandra was used at Twitter[0] to store quite a lot of time series data. A typical production instance of the time series database is based on four distinct Cassandra clusters, each responsible for a different dimension (real-time, historical, aggregate, index) due to different performance constraints. These clusters are amongst the largest Cassandra clusters deployed in production today and account for over 500 mi…

Believe they've moved to Manhattan, their own custom datastore:

https://blog.twitter.com/2014/manhattan-our-real-time-multi-...

Re: Ask HN: What DB to use for huge time series?

#82

Post by yummyfajitas »

sciurus wrote: post #78
yummyfajitas wrote: post #66

Graphite is a mature system. It's a pain in the ass, but I generally find it essential for server monitoring. I'm working on a timeseries database aimed at replacing graphite. It's just getting started, so it probably won't work immediately, but contributions are welcome. Currently the write performance is already better than graphite [1]. https://github.com/stucchio/timeserieszen [1] This was one of the design goals…

Cool project! I commented on some other graphite replacement projects at https://news.ycombinator.com/item?id=8368689

I didn't realize graphite was officially dead. I must say, however, that it was the shittiest piece of software I've ever relied on and loved.

Re: Ask HN: What DB to use for huge time series?

#83

Post by angersock »

BWStearns wrote: post #40
ddlatham wrote: post #27

For a good answer, you need to provide a lot more detail in the requirements: - What do the writes look like? If they are coming in a stream how many writes per second do you need to support? If they are a bulk load how large and frequent are the batches? Simple numerical values? - What do the reads look like? How many queries per second do you need to support? How much data per query? How fast do the queries need to…

Part of the reason the question was light on details is that this is just at the very beginning and a lot of relevant things aren't locked in yet. Below are the back of napkin results and are subject to the risk of being laughably wrong. Writes: not totally sure in terms of how the data is being packaged before being sent yet, but it'll probably be more than 10 writes a second but less than 1000 initially(?). Not sur…

What sort of data are you collecting?

Re: Ask HN: What DB to use for huge time series?

#84

Post by geocar »

chollida1 wrote: post #41

KDB+ http://kx.com/kdb-plus.php I have no affiliation, other than being a customer. Its as close to a standard as you can find in finance. There are many useful tutorials out there that let you try it out and you can usually get an eval version to try before you buy. http://code.kx.com/wiki/Startingkdbplus/contents If you find something that is comparable in terms of performance and features, but cheaper, please mail…

I'd also recommend KDB+

I find my code very clear and readable in it.

Re: Ask HN: What DB to use for huge time series?

#85

Post by sdab »

BWStearns wrote: post #74
sdab wrote: post #69

Earlier quoted context omitted.

In this case, I would still recommend Cassandra. It can easily handler the data sizes you mention as well as the write rates you imply further down the thread. Cassandra has a nice and simple architecture (every node is identical, no zookeeper roles etc), high write performance and scalability [1], and is fairly robust. My main piece of advice is to get the tables correctly set up. You need to know exactly what queri…

Thanks a ton. I am leaning towards a solution that involves Cassandra. What would you say about using something on top of it like Blueflood?

I havent used Blueflood, so I couldnt say but it looks like an interesting project.

Re: Ask HN: What DB to use for huge time series?

#86

Post by general_failure »

yummyfajitas wrote: post #82
sciurus wrote: post #78

Earlier quoted context omitted.

Cool project! I commented on some other graphite replacement projects at https://news.ycombinator.com/item?id=8368689

I didn't realize graphite was officially dead. I must say, however, that it was the shittiest piece of software I've ever relied on and loved.

I didn't realize this either. In fact, we just moved to graphite :-(

Re: Ask HN: What DB to use for huge time series?

#87

Post by siganakis »

bkeroack wrote: post #55
chollida1 wrote: post #41

KDB+ http://kx.com/kdb-plus.php I have no affiliation, other than being a customer. Its as close to a standard as you can find in finance. There are many useful tutorials out there that let you try it out and you can usually get an eval version to try before you buy. http://code.kx.com/wiki/Startingkdbplus/contents If you find something that is comparable in terms of performance and features, but cheaper, please mail…

Look at the source. (c) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c/k.h (c#) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c.cs This guy is truly depraved.

Remember that KDB is based on K, which stems from APL, which relies on symbols rather than words for its functions.

Coming from that background, C and especially C# must seem extremely verbose.

For example (from Wikipedia):

In K, finding the prime numbers from 1 to R is done with [0]:

    (!R)@&{&/x!/:2_!x}'!R
And APL[1]:

    (~R∊R∘.×R)/R←1↓ιR
Its truly awesome stuff.

[0]: http://en.wikipedia.org/wiki/K_(programming_language) [1]: http://en.wikipedia.org/wiki/APL_(programming_language)

Re: Ask HN: What DB to use for huge time series?

#88

Post by [deleted] »

bkeroack wrote: post #55
chollida1 wrote: post #41

KDB+ http://kx.com/kdb-plus.php I have no affiliation, other than being a customer. Its as close to a standard as you can find in finance. There are many useful tutorials out there that let you try it out and you can usually get an eval version to try before you buy. http://code.kx.com/wiki/Startingkdbplus/contents If you find something that is comparable in terms of performance and features, but cheaper, please mail…

Look at the source. (c) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c/k.h (c#) http://code.kx.com/wsvn/code/kx/kdb%2B/c/c.cs This guy is truly depraved.

[deleted]

Re: Ask HN: What DB to use for huge time series?

#89

Post by Diederich »

ad_hominem wrote: post #30

I have a timeseries problem on the backburner, and like you am hopeful for InfluxDB but it's still missing a couple features that I need, so haven't used it yet. As another person mentioned, you're going to be looking at columnar databases (few/one rows, with a very large amount of columns) if you have truly large storage requirements. Since my data is still small, I'm sticking with Postgres for now. I've seen a coup…

What features are you from InfluxDB? I am a long-time graphite user, and I just saw InfluxDB, and it looked really good.

Re: Ask HN: What DB to use for huge time series?

#90

Post by siliconviking »

new_test wrote: post #33
siliconviking wrote: post #15

First question - do you even need a database right now? Have you for example considered using CSV files and simply loading those files into Pandas or R on demand? I am currently working on a project analyzing massive amounts of options data and have found this approach to be both quite easy as well as flexible to work with... and as my project matures I may move select parts of it into a database.

>loading those files into Pandas or R What is "massive" for you? I was under impression you can't use R or pandas for anything that doesn't fit into memory.

As for massive - something like daily options data for 3000 stocks, spanning a number of years, with information down to the tranche level (let's say 60 million rows if stored in a relational database fashion). In my case the analysis can be done on the stock level though, which means that only a 3000th of the dataset needs to be loaded into memory at any time.
Post reply on HN