Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on Time-series Databases

jmoiron.net

Post reply on HN
Print view View original

41–50 of 132 posts

Re: Thoughts on Time-series Databases

#41

Post by Mahn »

olviko wrote: post #3

What are the benefits of introducing specialized time-series databases vs using Redis, Casandra, or some Sql database?

InfluxDB can downsample on the fly, to name an example. Functionality like this out of the box can be pretty handy, otherwise you have to build it yourself.

Re: Thoughts on Time-series Databases

#42

Post by robochat »

tsdbase wrote: post #25
Everlag wrote: post #12

I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…

I've hit the same problem and I would like to move back to a SQL data store. However none of the nice dashboards / visualizations support postgres or any SQL database (for now)... My question (to everyone): what do you use as replacement for kibana or grafana?

I've just implemented a custom backend for graphite-api which seems to be working ok although I don't have crazy requirements. https://github.com/brutasse/graphite-api is a cleaned up fork of graphite (which is much easier to install). I'm using grafana as the front-end and my data is in a postgresql database and graphite-api is linking them together.

Re: Thoughts on Time-series Databases

#43

Post by rw »

rdtsc wrote: post #37
Everlag wrote: post #12

I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…

I evaluated InfluxDB for an advanced packet capture and processing application and it couldn't handle things very well. Namely expiry of old data, blocking too much on inserts. So I wrote my own in Python + C extensions. It turned out well. Has been going non-stop for year and a half now.

What data structures does it use?

Re: Thoughts on Time-series Databases

#44

Post by cngupte »

We have been using Blueflood (https://github.com/rackerlabs/blueflood) in production for almost 3 years now and operate it at scale supporting ingestion rate of around 5 million metrics/min and performance tested it up to 16 million. We have even created a product around it http://docs.rackspace.com/cmet/api/v1.0/cmet-gettingstarted/... over here at Rackspace. Amongst the various cool features, the worth mentioning ones are graphite/grafana integration which allow you to use blueflood as a backend for plotting metrics and integrations with various metrics collection tools like StatsD, Collectd etc.

Re: Thoughts on Time-series Databases

#46

Post by engi_nerd »

maaku wrote: post #45
engi_nerd wrote: post #30

Earlier quoted context omitted.

Aircraft telemetry.

You really need to know the position of an aircraft to a thousandth of a second?

I said nothing about position. But a lot of things measured have to do with how the structure of the aircraft responds to turbulence, rough air, and aeroacoustic vibration (aka flutter). So there might be modes where structural components have harmonics that are pretty high (several hundred to over 1000 Hz). Therefore you must use a transducer that has a frequency response that can cover that range, and sample the output of the transducer at least twice that rate (at an absolute theoretical minimum, but rule of thumb is 5x oversampling).

Re: Thoughts on Time-series Databases

#47

Post by LukeHoersten »

Everlag wrote: post #12

I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…

It'd be great to see a more detailed guide to using and tuning PGSQL for use as a TS DB.

Re: Thoughts on Time-series Databases

#48

Post by akavel »

maaku wrote: post #45
engi_nerd wrote: post #30

Earlier quoted context omitted.

Aircraft telemetry.

You really need to know the position of an aircraft to a thousandth of a second?

Possibly he might measure some other thing, but still after your question I started to wonder how much a passenger plane moves per thousandth of a second.

So: some quick googling suggests that "economical cruising speed" of an Airbus A320 is 840km/h [1]. After quick back-of-envelope calculations, this gives ~230m/s, so 0.2m per 0.001s. Given some possible uncertainty of a single measurement, I'd imagine that's not unreasonable level of precision when e.g. your Airbus is landing on an airport.

[1]: http://www.airliners.net/aircraft-data/stats.main?id=23

Re: Thoughts on Time-series Databases

#49

Post by sciurus »

gtrubetskoy wrote: post #33

The subject of Time Series has lately been on my mind as well, see my blog posts on accuracy of Graphite vs RRD, as well as InfluxDB storage: http://grisha.org/ I am leaning towards none of the above being the best solution and am in the process of writing my own (too early to announce yet).

Like I mentioned here [0], whisper isn't the only storage option for graphite. Another user [1] mentioned blueflood. Have you evaluated any of these cassandra-based options?

[0] https://news.ycombinator.com/item?id=9808035

[1] https://news.ycombinator.com/item?id=9808662

Re: Thoughts on Time-series Databases

#50

Post by sleepydog »

tsdbase wrote: post #25
Everlag wrote: post #12

I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…

I've hit the same problem and I would like to move back to a SQL data store. However none of the nice dashboards / visualizations support postgres or any SQL database (for now)... My question (to everyone): what do you use as replacement for kibana or grafana?

If I decided to move to using a SQL data store (I use graphite now), I would re-implement the graphite API as a listener process, or write a graphite backend. The biggest strength of Graphite is how simple it is to upload and query metrics, and I wouldn't want to lose that even if the backend were to change.
Post reply on HN