It’s About Time for Time Series Databases

#1

Post by okket » Thu, Jan 25, 2018, 12:52 PM UTC

It’s About Time for Time Series Databases
nextplatform.com

Re: It’s About Time for Time Series Databases

#2

Post by RegBarclay » Thu, Jan 25, 2018, 1:34 PM UTC

How is TimescaleDB different from OSIsoft PI?

Re: It’s About Time for Time Series Databases

#3

Post by arbesfeld » Thu, Jan 25, 2018, 1:40 PM UTC

We use Cassandra extensively at https://logrocket.com. How does performance compare vs Cassandra when you don't need the semantics or transactions of SQL? I'm surprised the article only provides benchmarks against PostgreSQL.

Re: It’s About Time for Time Series Databases

#4

Post by kodablah » Thu, Jan 25, 2018, 1:44 PM UTC

Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech company. Sorry TimescaleDB, I see you have raised a decent amount of funding, but I have to choose my DBs w/ trepidation these days.

0 - https://www.theregister.co.uk/2017/07/13/will_the_last_perso...

Re: It’s About Time for Time Series Databases

#5

Post by stdbrouw » Thu, Jan 25, 2018, 1:50 PM UTC

> nobody wants to have large grain snapshots of data for any dataset that is actually comprised of a continuous stream of data points

Except, of course, for those who realize that the precision of a statistic only increases at sqrt(n) and that a biased dataset will remain biased regardless of how much data you have. I'll take a large grain dataset that I can load on my computer and analyze in five minutes over a finer grained dataset where I need to set up a cluster before I can even get started. Enough with the "let's store everything" fetishism already.

(Somewhat tangential to the blog post, I realize.)

Re: It’s About Time for Time Series Databases

#6

Post by walterbell » Thu, Jan 25, 2018, 1:51 PM UTC

kodablah wrote: post #4

Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…

Is the concern mitigated by this being an open-source Postgres extension? One question would be whether they have attracted external OSS contributors.

Re: It’s About Time for Time Series Databases

#7

Post by shangxiao » Thu, Jan 25, 2018, 1:55 PM UTC

This is a bit easier to grok: http://www.timescale.com/how-it-works

(if you're new to this like me)

Re: It’s About Time for Time Series Databases

#8

Post by YCode » Thu, Jan 25, 2018, 1:57 PM UTC

> Back in 2014...developers...could use relational databases with SQL interfaces, which are easy to use but they don’t scale well.

I'm skeptical.

Re: It’s About Time for Time Series Databases

#9

Post by teej » Thu, Jan 25, 2018, 2:08 PM UTC

stdbrouw wrote: post #5

> nobody wants to have large grain snapshots of data for any dataset that is actually comprised of a continuous stream of data points Except, of course, for those who realize that the precision of a statistic only increases at sqrt(n) and that a biased dataset will remain biased regardless of how much data you have. I'll take a large grain dataset that I can load on my computer and analyze in five minutes over a fine…

The reason we need to store everything is less about needing perfect accuracy of measurement (though I think we do want it) and more about the curse of dimensionality[0]. We want to slice, pivot, and filter datasets more aggressively than ever before which helps drive aggressive data collection.

[0] - https://en.wikipedia.org/wiki/Curse_of_dimensionality

Re: It’s About Time for Time Series Databases

#10

Post by apohn » Thu, Jan 25, 2018, 2:09 PM UTC

RegBarclay wrote: post #2

How is TimescaleDB different from OSIsoft PI?

Not just OSISoft PI, but there are many other historians for time series data.

https://en.wikipedia.org/wiki/Operational_historian

At least in my experience, historians are rarely recommend for complex or ad-hoc queries. Typically you just pull the data (by tags) into another application and do your data processing there. It looks like in timeseriesdb lets you execute complex queries in the database. Historians typically only let you fetch data by tag and you need a metadata (e.g. asset management) framework on top to organize the data (e.g. give me avg temp every 5 minutes by sensor). It looks like with timeseriesdb you can have strings/text as fields within the timeseries table, which removes the need (to some degree) to join the data with a seperate metadata database.

I've also never heard of anybody using these commercial historians for time series data you'd see from non industrial processes (e.g. stock data, price tracking, GPS location of people or moving assets, time between clicks on a website,etc).

All that being said, OSISoft PI and AF have their warts, but OSISoft has been around for a while and PI has been battle tested in various industries (e.g. Oil & Gas, Manufacturing). It's closed source and you have to pay for it, so it's probably not attractive to startups and smaller companies. But it does come with a support organization if you need it and can pay for it. And IME data retrieval from PI is extremely performant!