Thoughts on Time-series Databases • Hacker News Board

Re: Thoughts on Time-series Databases

#21

Post by philsnow » Tue, Jun 30, 2015, 8:38 PM UTC

Everlag wrote: post #12

I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…

> influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling

> to [InfluxDB's] credit it barely moved beyond idle resource usage when I was stuffing it full of data.

solution: always be cramming it full of data ?

I don't see how it could idle when you're bulk writing, but when it's not serving any traffic it's taking up 50% of cpu (unless it's doing some kind of indexing or cleanup or something in the background, but you would expect that to eventually quiesce).

Re: Thoughts on Time-series Databases

#22

Post by bbrazil » Tue, Jun 30, 2015, 8:44 PM UTC

Prometheus uses a file per timeseries, with two levels of delta encoding to keep the data small. This is our second major storage iteration, and seems to be doing pretty well with a single server able to handle over 2M timeseries.

See http://prometheus.io/docs/introduction/faq/#why-does-prometh...

Re: Thoughts on Time-series Databases

#23

Post by olviko » Tue, Jun 30, 2015, 8:56 PM UTC

Everlag wrote: post #12

I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…

Same here. InfluxDb has a pretty nice DSL. I wish they switched their backend to something usable and mature instead of re-inventing the bicycle. Operationally, it is expensive to support specialized databases like Influx, unless it is your core business, I guess...

Re: Thoughts on Time-series Databases

#24

Post by olviko » Tue, Jun 30, 2015, 9:02 PM UTC

_lbaq wrote: post #8

olviko wrote: post #3

What are the benefits of introducing specialized time-series databases vs using Redis, Casandra, or some Sql database?

Very few if any if your talking Cassandra (I would never store time series in redis), I worked developing TSD and are active in the space, most companies goes with Cassandra these days and builds computational frameworks upon that.

Just curious, why not Redis?

Re: Thoughts on Time-series Databases

#25

Post by tsdbase » Tue, Jun 30, 2015, 9:02 PM UTC

Everlag wrote: post #12

I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…

I've hit the same problem and I would like to move back to a SQL data store. However none of the nice dashboards / visualizations support postgres or any SQL database (for now)...

My question (to everyone): what do you use as replacement for kibana or grafana?

Re: Thoughts on Time-series Databases

#26

Post by bbrazil » Tue, Jun 30, 2015, 9:05 PM UTC

olviko wrote: post #3

What are the benefits of introducing specialized time-series databases vs using Redis, Casandra, or some Sql database?

One way to think of it is why would you use Cassandra when you already have a filesystem?

Those databases are quite raw, to be useful for a use case like monitoring you need to build on top of them to allow for ability to get data from agents, do some processing and enable efficient querying of the data.

Re: Thoughts on Time-series Databases

#27

Post by engi_nerd » Tue, Jun 30, 2015, 9:15 PM UTC

I'm always amused when I see the criterion for a "very dense" time series as data being collected more than once per second. In my business (telemetry), we often record parameters thousands of times per second, depending on what we are trying to measure.

Re: Thoughts on Time-series Databases

#28

Post by tsdbase » Tue, Jun 30, 2015, 9:18 PM UTC

engi_nerd wrote: post #27

I'm always amused when I see the criterion for a "very dense" time series as data being collected more than once per second. In my business (telemetry), we often record parameters thousands of times per second, depending on what we are trying to measure.

Can you be more specific? Is it car engine telemetry?

Re: Thoughts on Time-series Databases

#29

Post by endymi0n » Tue, Jun 30, 2015, 9:19 PM UTC

engi_nerd wrote: post #27

I'm always amused when I see the criterion for a "very dense" time series as data being collected more than once per second. In my business (telemetry), we often record parameters thousands of times per second, depending on what we are trying to measure.

Not that funny anymore if you try to record 1MM time series at once. :)

Re: Thoughts on Time-series Databases

#30

Post by engi_nerd » Tue, Jun 30, 2015, 9:22 PM UTC

tsdbase wrote: post #28

engi_nerd wrote: post #27

I'm always amused when I see the criterion for a "very dense" time series as data being collected more than once per second. In my business (telemetry), we often record parameters thousands of times per second, depending on what we are trying to measure.

Can you be more specific? Is it car engine telemetry?

Aircraft telemetry.