Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

pipelinedb.com

Post reply on HN
Print view View original

11–20 of 64 posts

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#11

Post by grammr »

skunkworker wrote: post #7

Interesting, this seems to be the other side of the postgres time series extension coin. TimescaleDB for writes, PipelineDB for reads.

I'm Derek, one of the co-founders--that's an interesting way to frame it, I think that makes a lot of sense at a high level.

We're in contact with the TSDB founders (awesome and super smart guys!) and are in the early stages of figuring out an integration that makes sense. That's most likely going to happen.

To anyone interested: we'd love to hear and consider your ideas re: TSDB integration. Feel free to open an issue in either repo (or add to an existing one) and tell us more!

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#12

Post by Fergi »

crescentfresh wrote: post #10

Looking over this cursorily, looks super cool. INSERT INTO events_stream (ts, value) VALUES (now(), '0ef346ac'); > As soon as the continuous view reads new incoming events and the distinct count is updated the reflect new information, the raw events will be discarded. So you create a table, insert into it, and it's always empty. Is that right? Does this work for any table in pg? How does pg know that the insert shoul…

This only applies to continuous views, not all PG tables. Think of continuous views in PipelineDB as very high throughput, incrementally updated materialized views. Raw data hits continuous queries in PipelineDB (continuous views) and only the output of the continuous queries is stored. So 1 billion events ingested could be distilled down into a single row that incrementally counts up from 1 => 1 billion as each data point arrives, instead of storing all of the 1 billion raw data points and counting them up later.

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#13

Post by grammr »

crescentfresh wrote: post #10

Looking over this cursorily, looks super cool. INSERT INTO events_stream (ts, value) VALUES (now(), '0ef346ac'); > As soon as the continuous view reads new incoming events and the distinct count is updated the reflect new information, the raw events will be discarded. So you create a table, insert into it, and it's always empty. Is that right? Does this work for any table in pg? How does pg know that the insert shoul…

I'm Derek, one of the co-founders--great questions!

> So you create a table, insert into it, and it's always empty. Is that right?

That is correct. Streams can only be read by continuous queries (e.g. you can't even run a SELECT on them).

> Does this work for any table in pg? How does pg know that the insert should NOT actually insert a row?

PipelineDB streams are represented as a specific kind of PostgreSQL foreign table [0], so only foreign tables created in a specific way will be considered streams. You can use triggers to write table rows and updates out to streams if you want to though.

[0] https://www.postgresql.org/docs/current/static/sql-createfor...

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#16

Post by skunkworker »

grammr wrote: post #11
skunkworker wrote: post #7

Interesting, this seems to be the other side of the postgres time series extension coin. TimescaleDB for writes, PipelineDB for reads.

I'm Derek, one of the co-founders--that's an interesting way to frame it, I think that makes a lot of sense at a high level. We're in contact with the TSDB founders (awesome and super smart guys!) and are in the early stages of figuring out an integration that makes sense. That's most likely going to happen. To anyone interested: we'd love to hear and consider your ideas re: TSDB integration. Feel free to open an iss…

That's great to hear, I'll be looking forward to seeing where those talks and collaborations go.

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#17

Post by scrollaway »

grammr wrote: post #11
skunkworker wrote: post #7

Interesting, this seems to be the other side of the postgres time series extension coin. TimescaleDB for writes, PipelineDB for reads.

I'm Derek, one of the co-founders--that's an interesting way to frame it, I think that makes a lot of sense at a high level. We're in contact with the TSDB founders (awesome and super smart guys!) and are in the early stages of figuring out an integration that makes sense. That's most likely going to happen. To anyone interested: we'd love to hear and consider your ideas re: TSDB integration. Feel free to open an iss…

Can you guys join forces and convince AWS to make both of those products available on RDS? :)

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#19

Post by grammr »

scrollaway wrote: post #17
grammr wrote: post #11

Earlier quoted context omitted.

I'm Derek, one of the co-founders--that's an interesting way to frame it, I think that makes a lot of sense at a high level. We're in contact with the TSDB founders (awesome and super smart guys!) and are in the early stages of figuring out an integration that makes sense. That's most likely going to happen. To anyone interested: we'd love to hear and consider your ideas re: TSDB integration. Feel free to open an iss…

Can you guys join forces and convince AWS to make both of those products available on RDS? :)

The most impactful thing you can do here is ask the RDS team for this. If enough users ask them for it they'll eventually begin seriously considering it :)

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#20

Post by scrollaway »

grammr wrote: post #19
scrollaway wrote: post #17

Earlier quoted context omitted.

Can you guys join forces and convince AWS to make both of those products available on RDS? :)

The most impactful thing you can do here is ask the RDS team for this. If enough users ask them for it they'll eventually begin seriously considering it :)

Already did!
Post reply on HN