Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on Time-series Databases

jmoiron.net

Post reply on HN
Print view View original

81–90 of 132 posts

Re: Thoughts on Time-series Databases

#81

Post by johngd »

sciurus wrote: post #14

If you are looking for time-series databases based on Cassandra that you can use with Graphite, check out Cyanite: https://github.com/pyr/cyanite and https://github.com/brutasse/graphite-cyanite KairosDB: https://github.com/kairosdb/kairosdb and https://github.com/kairosdb/kairos-carbon and https://github.com/Lastik/KairosdbGraphiteFinder

Have you used cyanite in any meaningful way? The original author's (pyr) repo has been pretty dead.

This person has been doing a lot of good work: https://github.com/mwmanley/cyanite

Re: Thoughts on Time-series Databases

#82

Post by engi_nerd »

nether wrote: post #61
engi_nerd wrote: post #52

Earlier quoted context omitted.

Ah, a fellow flight test person. I don't handle the warehousing of data, so my knowledge is limited. I know a database is involved and used but I could not say what database or provide any specifics. Sorry. EDIT: I would like to know how other flight test people visualize their test data. Seems like you can only make so many envelope charts and strip charts (they work and are effective, but man, sometimes I'd like so…

All we use are strip charts. It's all I saw at Boeing as well (through some proprietary charting software IIRC). What I'd like is some "rich" time series data format that stores a history of modifications (removing noise, filtering etc.), and an overall better interface for scrolling through the data (Windaq is pretty barebones).

Wow, that is pretty rough. That's so rough that I would consider figuring out some way to export your data from that software and then visualize it with python/matplotlib or similar.

Re: Thoughts on Time-series Databases

#83

Post by engi_nerd »

akavel wrote: post #48
maaku wrote: post #45

Earlier quoted context omitted.

You really need to know the position of an aircraft to a thousandth of a second?

Possibly he might measure some other thing, but still after your question I started to wonder how much a passenger plane moves per thousandth of a second. So: some quick googling suggests that "economical cruising speed" of an Airbus A320 is 840km/h [1]. After quick back-of-envelope calculations, this gives ~230m/s, so 0.2m per 0.001s. Given some possible uncertainty of a single measurement, I'd imagine that's not un…

I guess I should mention that GPS position is a perfectly normal thing to include in a telemetry stream.

Re: Thoughts on Time-series Databases

#84

Post by jboggan »

kvcc01 wrote: post #80
jnordwick wrote: post #72

Really? The big boy in the field KDB+ isn't mentioned? Kx's database is pretty much the gold standard for performance in time series, historical and real-time. http://kxcommunity.com/

I use kdb/Q at work and it’s a fun tool to play with so long as someone else is paying for it. It is quite common in finance (and comparably uncommon outside of it). It’s very expensive of course, and the learning curve is hard. In fact, there are plenty of businesses that have sprung up around kdb that offer consultancy services to help you get started. In an unusual maneuver, one of these consulting businesses actu…

32-bit version is free to play with nowadays: http://kx.com/software-download.php

Re: Thoughts on Time-series Databases

#85

Post by engi_nerd »

kbenson wrote: post #66
engi_nerd wrote: post #65

Earlier quoted context omitted.

Fair enough. In the telemetry world the second case you mention is rarely seen. I'm sure someone, somewhere is doing something like that, so I won't say never. In flight testing you have anywhere from a few measurements to thousands of measurements gathered from physical transducers, and then digital data bus information that can vary dramatically from platform to platform, from just a few kilobytes/sec to gigabytes/…

There's many, many cases where you have lots of discrete inputs but a lower time threshold. In/out bytes per port on switches in a datacenter. CPU/memory/disk stats on virts you spin up for on-demand load, or just your normal datacenter deployment. Employees/vehicle locations in the field. Often you don't even need per-second time granularity, but that can be made up for with a larger number of discrete inputs (how m…

Indeed, those cases are normal in other industries, but not in mine.

Re: Thoughts on Time-series Databases

#86

Post by Everlag »

martindale wrote: post #60
Everlag wrote: post #19

Earlier quoted context omitted.

It was pretty awful watching the vps choke to death when I tried implementing any feature using full set prices. Pegged at 99% cpu usage with the go garbage collector frantically trying to not let the process crash... that was not an environment I wanted to take to production.

> vps There's your problem. If you're at the "make it fast" part of "make it work, make it right, make it fast", you should almost certainly be on dedicated hardware.

That was actually the "make it work" portion: I was attempting to grab some 10 full set prices at a time for a gallery feature and managed to crash the influxdb.

Re: Thoughts on Time-series Databases

#87

Post by elektronaut »

engi_nerd wrote: post #27

I'm always amused when I see the criterion for a "very dense" time series as data being collected more than once per second. In my business (telemetry), we often record parameters thousands of times per second, depending on what we are trying to measure.

I've got a system here capable of collecting data close to a hundred thousand times per second from tens of sources, with perceptual real time (It's called multitrack audio recording, and commonly runs on your run of the mill laptop. HD video would probably be a few orders of magnitude more data than that, with even more processing.

Computers are really, really good at these kinds of things if only the software is efficient enough.

Re: Thoughts on Time-series Databases

#88

Post by nether »

engi_nerd wrote: post #82
nether wrote: post #61

Earlier quoted context omitted.

All we use are strip charts. It's all I saw at Boeing as well (through some proprietary charting software IIRC). What I'd like is some "rich" time series data format that stores a history of modifications (removing noise, filtering etc.), and an overall better interface for scrolling through the data (Windaq is pretty barebones).

Wow, that is pretty rough. That's so rough that I would consider figuring out some way to export your data from that software and then visualize it with python/matplotlib or similar.

I've been working on a Python module to read the files directly: https://github.com/choj/cldumpread/tree/master/WindaqReader/.... Scipy/numpy/matplotlib totally opens up what we can do with the data. The hard part is convincing a mechanical engineering manager who's extremely averse to new tools.

Re: Thoughts on Time-series Databases

#89

Post by Serow225 »

engi_nerd wrote: post #46
maaku wrote: post #45

Earlier quoted context omitted.

You really need to know the position of an aircraft to a thousandth of a second?

I said nothing about position. But a lot of things measured have to do with how the structure of the aircraft responds to turbulence, rough air, and aeroacoustic vibration (aka flutter). So there might be modes where structural components have harmonics that are pretty high (several hundred to over 1000 Hz). Therefore you must use a transducer that has a frequency response that can cover that range, and sample the ou…

The highest sample rates I remember seeing was 20kHz, for the pressure sensors used in turbofan inlet distortion testing.

Re: Thoughts on Time-series Databases

#90

Post by Serow225 »

nether wrote: post #61
engi_nerd wrote: post #52

Earlier quoted context omitted.

Ah, a fellow flight test person. I don't handle the warehousing of data, so my knowledge is limited. I know a database is involved and used but I could not say what database or provide any specifics. Sorry. EDIT: I would like to know how other flight test people visualize their test data. Seems like you can only make so many envelope charts and strip charts (they work and are effective, but man, sometimes I'd like so…

All we use are strip charts. It's all I saw at Boeing as well (through some proprietary charting software IIRC). What I'd like is some "rich" time series data format that stores a history of modifications (removing noise, filtering etc.), and an overall better interface for scrolling through the data (Windaq is pretty barebones).

I remember talking to someone that had an in-house flight test data warehouse with these kind of features, I think it might have been Cessna - give their FT dept a ring :)
Post reply on HN