Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on Time-series Databases

jmoiron.net

Post reply on HN
Print view View original

61–70 of 132 posts

Re: Thoughts on Time-series Databases

#61

Post by nether »

engi_nerd wrote: post #52
nether wrote: post #51

Earlier quoted context omitted.

Do you use a database for your data? We store our flight test data as individual DATAQ files with a naming scheme. This is manageable for our very simple civil aircraft so far.

Ah, a fellow flight test person. I don't handle the warehousing of data, so my knowledge is limited. I know a database is involved and used but I could not say what database or provide any specifics. Sorry. EDIT: I would like to know how other flight test people visualize their test data. Seems like you can only make so many envelope charts and strip charts (they work and are effective, but man, sometimes I'd like so…

All we use are strip charts. It's all I saw at Boeing as well (through some proprietary charting software IIRC). What I'd like is some "rich" time series data format that stores a history of modifications (removing noise, filtering etc.), and an overall better interface for scrolling through the data (Windaq is pretty barebones).

Re: Thoughts on Time-series Databases

#62

Post by kbenson »

engi_nerd wrote: post #27

I'm always amused when I see the criterion for a "very dense" time series as data being collected more than once per second. In my business (telemetry), we often record parameters thousands of times per second, depending on what we are trying to measure.

The collection time is only a single dimension, to see the scope you also need to know how many different pieces of data are being collected and for how many discrete groupings. Compare 100 inputs for a single device 1000 times a second to 100 inputs times 1000 devices once a second.

Re: Thoughts on Time-series Databases

#63

Post by MichaelGG »

rdtsc wrote: post #11

> Aggregation and expiry start to look a lot like dimensionality: they can be implemented asynchronously in a separate policy layer.It doesn't seem important that the actual storage engine is mindful of these; in fact, it's probably better ignored for efficiency's sake. I wrote my own time series database and actually having well working expiry was harder than expected. I sharded each one of the files into 2GB or 24h…

What industry or app was this for? I wrote something that sounds like this for VoIP packet capture. Though the rates were around 5TB/day. I'd keep the latest indexes in RAM before delta-encoding and writing out. Periodically, merge multiple indexes into one.

Worked surprisingly well, and the indexing overhead was only a few bits per packet (given enough similarity in packets over a short chunk of time (say, 1 minute), then just indexing unique values per chunk -- it was all fairly efficient.)

Re: Thoughts on Time-series Databases

#64

Post by olidb2 »

tsdbase wrote: post #25
Everlag wrote: post #12

I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…

I've hit the same problem and I would like to move back to a SQL data store. However none of the nice dashboards / visualizations support postgres or any SQL database (for now)... My question (to everyone): what do you use as replacement for kibana or grafana?

Shameless plug here - The author of the article does build time-series databases for a living, and more specifically the Datadog monitoring platform - which will gladly collect your millions of metrics, graph them and alert on them, along with all the events you care to keep :-)

We've been through a number of data stores ourselves, starting with Postgres back in 2010 - then on to Redis + Cassandra before we built our own. But that's a story for another post...

http://datadog.com

Re: Thoughts on Time-series Databases

#65

Post by engi_nerd »

kbenson wrote: post #62
engi_nerd wrote: post #27

I'm always amused when I see the criterion for a "very dense" time series as data being collected more than once per second. In my business (telemetry), we often record parameters thousands of times per second, depending on what we are trying to measure.

The collection time is only a single dimension, to see the scope you also need to know how many different pieces of data are being collected and for how many discrete groupings. Compare 100 inputs for a single device 1000 times a second to 100 inputs times 1000 devices once a second.

Fair enough. In the telemetry world the second case you mention is rarely seen. I'm sure someone, somewhere is doing something like that, so I won't say never. In flight testing you have anywhere from a few measurements to thousands of measurements gathered from physical transducers, and then digital data bus information that can vary dramatically from platform to platform, from just a few kilobytes/sec to gigabytes/sec.

Re: Thoughts on Time-series Databases

#66

Post by kbenson »

engi_nerd wrote: post #65
kbenson wrote: post #62

Earlier quoted context omitted.

The collection time is only a single dimension, to see the scope you also need to know how many different pieces of data are being collected and for how many discrete groupings. Compare 100 inputs for a single device 1000 times a second to 100 inputs times 1000 devices once a second.

Fair enough. In the telemetry world the second case you mention is rarely seen. I'm sure someone, somewhere is doing something like that, so I won't say never. In flight testing you have anywhere from a few measurements to thousands of measurements gathered from physical transducers, and then digital data bus information that can vary dramatically from platform to platform, from just a few kilobytes/sec to gigabytes/…

There's many, many cases where you have lots of discrete inputs but a lower time threshold. In/out bytes per port on switches in a datacenter. CPU/memory/disk stats on virts you spin up for on-demand load, or just your normal datacenter deployment. Employees/vehicle locations in the field. Often you don't even need per-second time granularity, but that can be made up for with a larger number of discrete inputs (how many trucks does UPS have in service?)

Re: Thoughts on Time-series Databases

#68

Post by jsmthrowaway »

sciurus wrote: post #49
gtrubetskoy wrote: post #33

The subject of Time Series has lately been on my mind as well, see my blog posts on accuracy of Graphite vs RRD, as well as InfluxDB storage: http://grisha.org/ I am leaning towards none of the above being the best solution and am in the process of writing my own (too early to announce yet).

Like I mentioned here [0], whisper isn't the only storage option for graphite. Another user [1] mentioned blueflood. Have you evaluated any of these cassandra-based options? [0] https://news.ycombinator.com/item?id=9808035 [1] https://news.ycombinator.com/item?id=9808662

At Foursquare, work was being done to put Hypertable under Graphite. It was incredible to use (my year+ queries returned in tens of milliseconds), but I don't know what came of it. Hypertable is criminally overlooked in the industry, and TSDB is a killer app; a little bit of glue code and you've basically invented a crude clone of Google's monitoring, with that stack.

Re: Thoughts on Time-series Databases

#69

Post by gane5h »

We store our event stream data in Elasticsearch. Two features that made it appealing:

  * the ingest-side can be scaled up by adding more shards
  * the query-side can be scaled up by adding more replicas
To compute rollup analytics, we make heavy use of Elasticsearch's aggregation framework to compute daily/weekly/monthly/quarterly active users.

From my understanding Postgres has many of these features, but the distributed features of ES are killer!

Re: Thoughts on Time-series Databases

#70

Post by 15155 »

LukeHoersten wrote: post #47
Everlag wrote: post #12

I actually just migrated 20 million rows of Magic: the Gathering price data from influxDB to postgres this week. For a few days of effort, I decreased my query latency by an order of a magnitude; a full set query, roughly 270 cards, went from 30 to 3 seconds with a cold cache. The migration was prompted by influxDB 0.8 eating 50% of the VPS' cpu and 77% of the ram while idling. It had no capability to index along any…

It'd be great to see a more detailed guide to using and tuning PGSQL for use as a TS DB.

As an addendum, I'd like to be able to do time-based append-only operations in Postgres, a la Datomic.

It pains me to UPDATE and DELETE, thereby destroying useful data.

Post reply on HN