Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on Time-series Databases

jmoiron.net

Post reply on HN
Print view View original

31–40 of 132 posts

Re: Thoughts on Time-series Databases

#31

Post by engi_nerd »

endymi0n wrote: post #29
engi_nerd wrote: post #27

I'm always amused when I see the criterion for a "very dense" time series as data being collected more than once per second. In my business (telemetry), we often record parameters thousands of times per second, depending on what we are trying to measure.

Not that funny anymore if you try to record 1MM time series at once. :)

I'm not anywhere in that realm because I don't have a data system that could support such a large amount of data. But then, I also have a physical system that does not need observations of 1 million distinct things (at least, not yet).

Re: Thoughts on Time-series Databases

#32

Post by fweespeech »

olviko wrote: post #24
_lbaq wrote: post #8

Earlier quoted context omitted.

Very few if any if your talking Cassandra (I would never store time series in redis), I worked developing TSD and are active in the space, most companies goes with Cassandra these days and builds computational frameworks upon that.

Just curious, why not Redis?

I'm not him obviously but...

If you are at the scale you can't dump it on just 1-2 nodes and call it a day [which is when you are start looking at Cassandra or a dedicated TSD] ...you really need 3 DC availability usually and Redis simply cannot do that in any reasonable way.

Re: Thoughts on Time-series Databases

#33

Post by gtrubetskoy »

The subject of Time Series has lately been on my mind as well, see my blog posts on accuracy of Graphite vs RRD, as well as InfluxDB storage: http://grisha.org/

I am leaning towards none of the above being the best solution and am in the process of writing my own (too early to announce yet).

Re: Thoughts on Time-series Databases

#34

Post by gtrubetskoy »

Everlag wrote: post #12

I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…

I've found PostgreSQL to be extremely fast if you store time series in arrays (http://www.postgresql.org/docs/9.4/static/arrays.html) in a round-robin fashion. You can also limit the array size, so that you have a fixed number of points per table row (thereby splitting your series across multiple rows), and if you adjust it such that it fits on one PG page it is quite performant.

Re: Thoughts on Time-series Databases

#35

Post by scott_karana »

philsnow wrote: post #21
Everlag wrote: post #12

I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…

> influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling > to [InfluxDB's] credit it barely moved beyond idle resource usage when I was stuffing it full of data. solution: always be cramming it full of data ? I don't see how it could idle when you're bulk writing, but when it's not serving any traffic it's taking up 50% of cpu (unless it's doing some kind of indexing or cleanup or something in the ba…

I think he meant that idle CPU (50%) and full throttle usage (51%) were hardly different? Who knows. :P

Re: Thoughts on Time-series Databases

#36

Post by Mahn »

Everlag wrote: post #12

I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…

Just for the record, InfluxDB 0.9 seems to actually be production ready now. Though it doesn't look there's an easy way to migrate to it yet from 0.8.

Re: Thoughts on Time-series Databases

#37

Post by rdtsc »

Everlag wrote: post #12

I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…

I evaluated InfluxDB for an advanced packet capture and processing application and it couldn't handle things very well. Namely expiry of old data, blocking too much on inserts. So I wrote my own in Python + C extensions. It turned out well. Has been going non-stop for year and a half now.

Re: Thoughts on Time-series Databases

#38

Post by nemothekid »

fweespeech wrote: post #32
olviko wrote: post #24

Earlier quoted context omitted.

Just curious, why not Redis?

I'm not him obviously but... If you are at the scale you can't dump it on just 1-2 nodes and call it a day [which is when you are start looking at Cassandra or a dedicated TSD] ...you really need 3 DC availability usually and Redis simply cannot do that in any reasonable way.

Even if you don't need a dedicated TSD with redundancy - Redis would still be more expensive to run given that you you would need to keep everything in memory. Given that you won't read 90% of the data most of the time, it makes little sense to store all of it in memory

Re: Thoughts on Time-series Databases

#40

Post by Mahn »

tsdbase wrote: post #25
Everlag wrote: post #12

I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…

I've hit the same problem and I would like to move back to a SQL data store. However none of the nice dashboards / visualizations support postgres or any SQL database (for now)... My question (to everyone): what do you use as replacement for kibana or grafana?

Did you consider a hybrid solution? You could store the most recent data in time series database for visualization purposes and dump the rest into a traditional SQL data store. Other than that, IIRC Grafana had plans for PostgresSQL but it's not there yet.
Post reply on HN