It’s About Time for Time Series Databases • Hacker News Board

Re: It’s About Time for Time Series Databases

#11

Post by Radim » Thu, Jan 25, 2018, 2:15 PM UTC

Perhaps not as lofty goal as time series, but a good, commercial database for vectors (aka the building representation block of modern machine learning) is long overdue too.

A few open source options exist (Spotify's Annoy, Facebook's FAISS, NMSLIB) but these are rather low-level, more hobby projects than enterprise level engines (index management, transactions, sharding…).

After building a few document similarity engines for our clients we took up the gauntlet and created ScaleText, https://scaletext.ai. It's still early days but the demand for a sane, scalable, cross-vertical and well-supported NLP technology is encouraging.

Re: It’s About Time for Time Series Databases

#12

Post by ryanworl » Thu, Jan 25, 2018, 2:16 PM UTC

teej wrote: post #9

stdbrouw wrote: post #5

> nobody wants to have large grain snapshots of data for any dataset that is actually comprised of a continuous stream of data points Except, of course, for those who realize that the precision of a statistic only increases at sqrt(n) and that a biased dataset will remain biased regardless of how much data you have. I'll take a large grain dataset that I can load on my computer and analyze in five minutes over a fine…

The reason we need to store everything is less about needing perfect accuracy of measurement (though I think we do want it) and more about the curse of dimensionality[0]. We want to slice, pivot, and filter datasets more aggressively than ever before which helps drive aggressive data collection. [0] - https://en.wikipedia.org/wiki/Curse_of_dimensionality

You can use sampling to both store every dimension of a data point and to not store an unwieldy amount of data.

Re: It’s About Time for Time Series Databases

#13

Post by gaius » Thu, Jan 25, 2018, 2:17 PM UTC

kodablah wrote: post #4

Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…

If Timescale goes bust you still have all your data in Postgres, which isn’t going anywhere. The risk is much lower.

Re: It’s About Time for Time Series Databases

#14

Post by kodablah » Thu, Jan 25, 2018, 2:18 PM UTC

walterbell wrote: post #6

kodablah wrote: post #4

Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…

Is the concern mitigated by this being an open-source Postgres extension? One question would be whether they have attracted external OSS contributors.

Depends on your reliance I suppose. Appears most of TimescaleDB's vendor-specific pieces are in the DDL area, but there are plenty of DML-specific extensions. I'm not saying TimescaleDB might not get to a level of maturity where I'd use it, I personally just can't justify it today. I've found DBs and ops/orchestration engines to be the most painful to move off from if you need to.

Re: It’s About Time for Time Series Databases

#15

Post by greenleafjacob » Thu, Jan 25, 2018, 2:20 PM UTC

kodablah wrote: post #4

Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…

Basho is in receivership, close to bankruptcy. https://www.theregister.co.uk/2017/07/31/end_of_the_road_for...

Re: It’s About Time for Time Series Databases

#16

Post by antirez » Thu, Jan 25, 2018, 2:26 PM UTC

kodablah wrote: post #4

Sorry to leave the technical detail part real quick. But is anyone else concerned about using a DB solely from a company built specifically around that DB? After Rethink DB (sustainability issue) and Foundation DB (bought and shuttered/hidden) and Riak (admittedly haven't kept up but I saw [0]), I am wary of using any DB that is not built by a large community or is not built as a non-core project from a large tech co…

What you say is another way to see the argument about OSS Project vs OSS licensed software which is actually a company and not a project.

Re: It’s About Time for Time Series Databases

#17

Post by tw1010 » Thu, Jan 25, 2018, 2:26 PM UTC

I have a hard time shaking the feeling that database innovation is being forced not by necessity, but by a culture of blind relentless change.

Re: It’s About Time for Time Series Databases

#18

Post by teej » Thu, Jan 25, 2018, 2:28 PM UTC

ryanworl wrote: post #12

teej wrote: post #9

Earlier quoted context omitted.

The reason we need to store everything is less about needing perfect accuracy of measurement (though I think we do want it) and more about the curse of dimensionality[0]. We want to slice, pivot, and filter datasets more aggressively than ever before which helps drive aggressive data collection. [0] - https://en.wikipedia.org/wiki/Curse_of_dimensionality

You can use sampling to both store every dimension of a data point and to not store an unwieldy amount of data.

This simply doesn’t work when you have sparsely populated dimensions and/or you don’t know what dimensions are important in advance. Both of these are very common. That’s why you don’t see a higher prevalence of estimated measurement.

Re: It’s About Time for Time Series Databases

#19

Post by RobAtticus » Thu, Jan 25, 2018, 2:35 PM UTC

arbesfeld wrote: post #3

We use Cassandra extensively at https://logrocket.com . How does performance compare vs Cassandra when you don't need the semantics or transactions of SQL? I'm surprised the article only provides benchmarks against PostgreSQL.

That is currently next in our pipeline for a benchmark blog post. Early results look good on both the read and write side in terms of raw performance, with the benefit of more complex queries being more easily expressable.

Re: It’s About Time for Time Series Databases

#20

Post by RobAtticus » Thu, Jan 25, 2018, 2:41 PM UTC

stdbrouw wrote: post #5

> nobody wants to have large grain snapshots of data for any dataset that is actually comprised of a continuous stream of data points Except, of course, for those who realize that the precision of a statistic only increases at sqrt(n) and that a biased dataset will remain biased regardless of how much data you have. I'll take a large grain dataset that I can load on my computer and analyze in five minutes over a fine…

Re: setting up a cluster

With TimescaleDB we've focused on single node performance to try and reduce the need for clustering. We've found performance scales very well just by adding more disk space if needed and more cores. So maybe some datasets are not practical for your laptop necessarily but a single instance on Azure/AWS/GCP is workable. No need for a cluster to get started :)

(Read scale out is available today and we are working on write scale out, hopefully later this year)