Big question for me: does it work on Heroku postgres?
PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL
51–60 of 64 posts
Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL
#52Running functions on top of the transaction log(in transaction order) is a really powerful thing.
Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL
#53How does this compare to Citus?
They are completely different products? Citus deals with scaling pipelinedb deals with continuous queries.
Citus advertises itself as an excellent way to achieve real-time analytics across billions of rows and tonnes of data, which this product also does.
How both products achieve this is however different.
Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL
#54Earlier quoted context omitted.
Thanks for the great summary, manigandham. We're actively working on the scale-out version of TimescaleDB that will allow you to transparently shard hypertables across many servers. Hope to announce more specifics in the next several months.
What is wrong with citus? Why reimplement it?
Use TimescaleDB if you have time-series data, if you want to scale out your OLTP workload then Citus is what you're looking for.
Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL
#55Earlier quoted context omitted.
What is wrong with citus? Why reimplement it?
Use TimescaleDB if you have time-series data, if you want to scale out your OLTP workload then Citus is what you're looking for.
Citus is also used for large time-series / analytics use cases e.g. https://www.citusdata.com/customers/heap
There's a question of what you actually want to do with the time-series data. If you don't expect to have much data or just want to store it and maybe occassionally query it, then a single server with partitioning (e.g. through pg_partman, Timescale) might be enough. If you want to build an analytical dashboard that needs to remain fast even if you're dealing with many users and terabytes of data per day, then you probably need Citus.
Citus can load, aggregate and query the data in parallel using all the cores in the cluster. It also supports Postgres' native partitioning and pg_partman: https://www.citusdata.com/blog/2018/01/24/citus-and-pg-partm...
Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL
#56Interesting, this seems to be the other side of the postgres time series extension coin. TimescaleDB for writes, PipelineDB for reads.
I'm Derek, one of the co-founders--that's an interesting way to frame it, I think that makes a lot of sense at a high level. We're in contact with the TSDB founders (awesome and super smart guys!) and are in the early stages of figuring out an integration that makes sense. That's most likely going to happen. To anyone interested: we'd love to hear and consider your ideas re: TSDB integration. Feel free to open an iss…
RE integration: A docker image with both TSDB and PipelineDB extensions and PostGIS, supporting PG11 ;) which is something I will look into doing myself, but lack the time to do so..
The time-series database of a project I'm on uses timescale and it's been great for the quick inserts and the `time_bucket` function has been very useful for aggregate queries.. But moving from aggregations generated on-the-fly to ones updated continuously on data change sounds like it could be awesome for us, so I am v happy to see this article today :-)
Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL
#57Big question for me: does it work on Heroku postgres?
No, not currently; however, we offer hosted deployments of PipelineDB as well as a SaaS product called Stride (stride.io), which is based on PipelineDB.
Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL
#58Is most of the intermediate processing done in memory, or is it limited by hd write speed?
I'm Derek, one of the co-founders--excellent question! The former. PipelineDB performs aggregations in memory on microbatches of events, and only merges the aggregate output of each microbatch with what's on disk. This is really the core idea behind why PipelineDB is so performant for continuous time-series aggregation. Microbatch size is configurable: http://docs.pipelinedb.com/conf.html .
Can you say a bit more about "performant" or point me to some information? I haven't found any yet. I'm processing millions of protobufs per second and would love to get away from batch jobs to do some incredibly basic counting -- this seems like a fit conceptually...If its a fit, any recommendations on the best way to get those protobufs off a kafka stream and into pipelinedb would be great, too!
Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL
#59Very interesting. Does it aggregate per day? If so, I wonder how it handles time-zone, I mean when to create a new day when you have agents on different time-zones.
Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL
#60Very interesting. Does it aggregate per day? If so, I wonder how it handles time-zone, I mean when to create a new day when you have agents on different time-zones.
How aggregations are performed are determined entirely by your own continuous view definitions [0]. In this case I'm guessing you'd want to include a time-based column in the aggregation GROUP BY clause.
And since PipelineDB is a PostgreSQL extension, you can use the timestamptz type (which includes timezone support), and in general you could pretty easily simply normalize your event timezones in your continuous view definitions. When you're reading aggregate data back out, you could cast the time-based column using whatever timezone the client prefers.
Thanks for the question--I hope that was helpful!