PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL • Hacker News Board

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#41

Post by akulkarni » Thu, Oct 25, 2018, 1:58 AM UTC

dkulchenko wrote: post #23

How does this compare to TimescaleDB? Are they solving the same problem in different ways or are they complementary projects? If it's the latter, what would that look like?

(Timescale founder) I'd say they are quite complementary. More here: https://news.ycombinator.com/item?id=18298004

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#42

Post by nwmcsween » Thu, Oct 25, 2018, 3:22 AM UTC

mfreed wrote: post #38

manigandham wrote: post #36

PipelineDB = Insert data with time component to be aggregated on the fly into always up-to-date summary tables using a variety of aggregation functions. Raw data is not persisted. TimescaleDB = Store data with time component into "hypertable" that is automatically partitioned by time, for faster queries when limited by time range. Single node and has helper methods to make time based bucketing and aggregation easier.…

Thanks for the great summary, manigandham. We're actively working on the scale-out version of TimescaleDB that will allow you to transparently shard hypertables across many servers. Hope to announce more specifics in the next several months.

What is wrong with citus? Why reimplement it?

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#43

Post by nwmcsween » Thu, Oct 25, 2018, 3:25 AM UTC

jadbox wrote: post #15

How does this compare to Citus?

They are completely different products? Citus deals with scaling pipelinedb deals with continuous queries.

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#44

Post by camel_gopher » Thu, Oct 25, 2018, 4:24 AM UTC

skunkworker wrote: post #7

Interesting, this seems to be the other side of the postgres time series extension coin. TimescaleDB for writes, PipelineDB for reads.

One of the challenges of managing time series data is to be able to manage both reads and writes at scale concurrently. I wrote about some of the ways different TSDBs approach this issue (Disclosure, I work for a commercial time series database provider)

https://www.irondb.io/2018/08/tsdbs-at-scale-part-two/

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#45

Post by crad » Thu, Oct 25, 2018, 4:31 AM UTC

manigandham wrote: post #36

PipelineDB = Insert data with time component to be aggregated on the fly into always up-to-date summary tables using a variety of aggregation functions. Raw data is not persisted. TimescaleDB = Store data with time component into "hypertable" that is automatically partitioned by time, for faster queries when limited by time range. Single node and has helper methods to make time based bucketing and aggregation easier.…

Check out pgxn.org for Postgres extension distribution /package management.

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#46

Post by tfehring » Thu, Oct 25, 2018, 5:12 AM UTC

manigandham wrote: post #36

PipelineDB = Insert data with time component to be aggregated on the fly into always up-to-date summary tables using a variety of aggregation functions. Raw data is not persisted. TimescaleDB = Store data with time component into "hypertable" that is automatically partitioned by time, for faster queries when limited by time range. Single node and has helper methods to make time based bucketing and aggregation easier.…

Minor clarification: do you actually need to choose TimescaleDB xor Citus? I thought the fact that both are Postgres extensions (as opposed to forks) meant that they could be used jointly.

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#47

Post by manigandham » Thu, Oct 25, 2018, 5:17 AM UTC

tfehring wrote: post #46

manigandham wrote: post #36

PipelineDB = Insert data with time component to be aggregated on the fly into always up-to-date summary tables using a variety of aggregation functions. Raw data is not persisted. TimescaleDB = Store data with time component into "hypertable" that is automatically partitioned by time, for faster queries when limited by time range. Single node and has helper methods to make time based bucketing and aggregation easier.…

Minor clarification: do you actually need to choose TimescaleDB xor Citus? I thought the fact that both are Postgres extensions (as opposed to forks) meant that they could be used jointly.

You can use both extensions on separate tables but they are not interoperable on the same table.

Timescale is limited to time-based partitioning on one node while Citus uses a master node with partitioning across multiple worker nodes. I don't see many situations why you would run both.

If you're just doing single-node time-series then use Timescale. If you have Citus then you can use the native partitions in 11 or pg_partman to add a secondary partitioning dimension.

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#48

Post by usgroup » Thu, Oct 25, 2018, 5:19 AM UTC

Fantastic guys, thank you! I’ve been looking forward to it becoming an extension for half a year. This is great news.

This basically means Postgres now has continuous views and a toolkbox of functions for running calculations. Combined with PG11 partitioning features and better parallel gusty execution, PG is an even more formidable choice for medium sized data.

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#49

Post by tpetry » Thu, Oct 25, 2018, 5:37 AM UTC

nwmcsween wrote: post #42

mfreed wrote: post #38

Earlier quoted context omitted.

Thanks for the great summary, manigandham. We're actively working on the scale-out version of TimescaleDB that will allow you to transparently shard hypertables across many servers. Hope to announce more specifics in the next several months.

What is wrong with citus? Why reimplement it?

Money isn‘t flowing to their pockets.

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#50

Post by Arqu » Thu, Oct 25, 2018, 6:43 AM UTC

I work closely in the space of providing time series databases as managed solutions. I can say that I am very happy to see this recent development of new tsd's and this with timescale is a huge bump to the industry/segment. Everybody currently measures some analytics and mostly user data and there is so much abuse with it, yet there is so much more you can measure and do and it is still very early stage. Farms, industrial applications, IoT and so much more. I'd love to just measure temperature and wind speed at unprecedented resolution.