I have a hard time shaking the feeling that database innovation is being forced not by necessity, but by a culture of blind relentless change.
Silicon Valley on reinventing the wheel: "Ours is rounder!"
It’s About Time for Time Series Databases
21–30 of 151 posts
Re: It’s About Time for Time Series Databases
#22Earlier quoted context omitted.
You can use sampling to both store every dimension of a data point and to not store an unwieldy amount of data.
This simply doesn’t work when you have sparsely populated dimensions and/or you don’t know what dimensions are important in advance. Both of these are very common. That’s why you don’t see a higher prevalence of estimated measurement.
You’re definitely not wrong, but if you simply can’t operate a system to store everything for some reason, sampling is a lot better than just doing an aggregation on one dimension and throwing everything else away.
For anyone reading this who is interest in a practical application with sampled operational data, check out Facebook Scuba.
https://research.fb.com/publications/scuba-diving-into-data-...
Re: It’s About Time for Time Series Databases
#23Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…
Basho is in receivership, close to bankruptcy. https://www.theregister.co.uk/2017/07/31/end_of_the_road_for...
For anyone just seeing this, its assets were purchased and are planned to be open-sourced: https://news.ycombinator.com/item?id=15182566
Re: It’s About Time for Time Series Databases
#24Earlier quoted context omitted.
You can use sampling to both store every dimension of a data point and to not store an unwieldy amount of data.
This simply doesn’t work when you have sparsely populated dimensions and/or you don’t know what dimensions are important in advance. Both of these are very common. That’s why you don’t see a higher prevalence of estimated measurement.
> you don’t know what dimensions are important in advance
But again this is a huge red flag. I've seen so many data science projects that started with "well, let's just get started with collecting everything and we will figure out what is important later on" and then spent so much time on infrastructure that no useful insights were ever produced.
Re: It’s About Time for Time Series Databases
#25Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…
Riak the database survives Basho the company. All assets (IP) were bought by bet365 (A large Basho customer) and made open source. Development continues.
/former Basho employee.
Re: It’s About Time for Time Series Databases
#26I have a hard time shaking the feeling that database innovation is being forced not by necessity, but by a culture of blind relentless change.
OK, but is that bad?
It might be bad for the specific organization pushing their new solutions, but for the software industry as a whole, it seems like the good side of "throw spaghetti against the wall and see what sticks". Many ideas will fail, but the good ones will stick around. So I'd say by all means, let people feel free to innovate. And those of us who consume the innovations just need to remember the mantra of "leading edge, not bleeding edge."
Re: It’s About Time for Time Series Databases
#27I have a hard time shaking the feeling that database innovation is being forced not by necessity, but by a culture of blind relentless change.
I have to say that a sentiment similar to this is why we built Timescale as a Postgres extension and not started a new database from scratch. We didn't want a new shiny thing just for kicks but rather a product focused on solving a particular problem.
Re: It’s About Time for Time Series Databases
#28How is TimescaleDB different from OSIsoft PI?
Not just OSISoft PI, but there are many other historians for time series data. https://en.wikipedia.org/wiki/Operational_historian At least in my experience, historians are rarely recommend for complex or ad-hoc queries. Typically you just pull the data (by tags) into another application and do your data processing there. It looks like in timeseriesdb lets you execute complex queries in the database. Historians typic…
Yeah, to underscore some differences more completely, timescale has:
- Full indexing and secondary index support.
- Support for transactional semantics
- Support for living along side relational data - including foreign keys to the relational data.
- Full trigger and constraint support
- Support for all of Postgres's native data types including JSON(B) and GIS location data.
Re: It’s About Time for Time Series Databases
#29This is a bit easier to grok: http://www.timescale.com/how-it-works (if you're new to this like me)
Re: It’s About Time for Time Series Databases
#30I have a hard time shaking the feeling that database innovation is being forced not by necessity, but by a culture of blind relentless change.
OK, but is that bad? It might be bad for the specific organization pushing their new solutions, but for the software industry as a whole, it seems like the good side of "throw spaghetti against the wall and see what sticks". Many ideas will fail, but the good ones will stick around. So I'd say by all means, let people feel free to innovate. And those of us who consume the innovations just need to remember the mantra…
I didn't say it was bad (though the tone of my post definitely hinted at it). I agree with you. Following the trail of necessity can easily lead to locally optimal points, away from the global maximum.