Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

pipelinedb.com

Post reply on HN
Print view View original

21–30 of 64 posts

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#21

Post by qaq »

scrollaway wrote: post #17
grammr wrote: post #11

Earlier quoted context omitted.

I'm Derek, one of the co-founders--that's an interesting way to frame it, I think that makes a lot of sense at a high level. We're in contact with the TSDB founders (awesome and super smart guys!) and are in the early stages of figuring out an integration that makes sense. That's most likely going to happen. To anyone interested: we'd love to hear and consider your ideas re: TSDB integration. Feel free to open an iss…

Can you guys join forces and convince AWS to make both of those products available on RDS? :)

So basically AWS will monetize something they have spent 0 resources building and will likely cannibalise the only viable monetization option?

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#22

Post by chatmasta »

qaq wrote: post #21
scrollaway wrote: post #17

Earlier quoted context omitted.

Can you guys join forces and convince AWS to make both of those products available on RDS? :)

So basically AWS will monetize something they have spent 0 resources building and will likely cannibalise the only viable monetization option?

Surely not, AWS has never done anything like that!

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#24

Post by ohnoesjmr »

Fergi wrote: post #12
crescentfresh wrote: post #10

Looking over this cursorily, looks super cool. INSERT INTO events_stream (ts, value) VALUES (now(), '0ef346ac'); > As soon as the continuous view reads new incoming events and the distinct count is updated the reflect new information, the raw events will be discarded. So you create a table, insert into it, and it's always empty. Is that right? Does this work for any table in pg? How does pg know that the insert shoul…

This only applies to continuous views, not all PG tables. Think of continuous views in PipelineDB as very high throughput, incrementally updated materialized views. Raw data hits continuous queries in PipelineDB (continuous views) and only the output of the continuous queries is stored. So 1 billion events ingested could be distilled down into a single row that incrementally counts up from 1 => 1 billion as each data…

You can't really do that with distinct, as if you have 1 billion distint entries, you essentially have to store all of them to dedup.

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#25

Post by [deleted] »

ohnoesjmr wrote: post #24
Fergi wrote: post #12

Earlier quoted context omitted.

This only applies to continuous views, not all PG tables. Think of continuous views in PipelineDB as very high throughput, incrementally updated materialized views. Raw data hits continuous queries in PipelineDB (continuous views) and only the output of the continuous queries is stored. So 1 billion events ingested could be distilled down into a single row that incrementally counts up from 1 => 1 billion as each data…

You can't really do that with distinct, as if you have 1 billion distint entries, you essentially have to store all of them to dedup.

[deleted]

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#26

Post by grammr »

ohnoesjmr wrote: post #24
Fergi wrote: post #12

Earlier quoted context omitted.

This only applies to continuous views, not all PG tables. Think of continuous views in PipelineDB as very high throughput, incrementally updated materialized views. Raw data hits continuous queries in PipelineDB (continuous views) and only the output of the continuous queries is stored. So 1 billion events ingested could be distilled down into a single row that incrementally counts up from 1 => 1 billion as each data…

You can't really do that with distinct, as if you have 1 billion distint entries, you essentially have to store all of them to dedup.

This is precisely why PipelineDB has rich support for data structures such as HyperLogLog [0]. HLL's allow you to track distincts information using fixed-size HLLs that only grow to about 14KB while encoding uniques counts for billions of distinct values. The tradeoff is about a ~0.8% margin of error, which users generally find acceptable.

Furthermore, PipelineDB has a special combine [1] aggregate that allows you to combine data structures such as HLL across multiple rows with no loss of information. A simpler example would be average: to get the actual average of multiple averages you obviously can't simply take the average of all the averages. Their weights must be taken into account, and combine handles that.

The capability to combine aggregate values in this way generalizes to all aggregates in PipelineDB.

[0] http://docs.pipelinedb.com/aggregates.html#hyperloglog-aggre...

[1] http://docs.pipelinedb.com/aggregates.html#combine

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#29

Post by grammr »

alakin wrote: post #28

Is most of the intermediate processing done in memory, or is it limited by hd write speed?

I'm Derek, one of the co-founders--excellent question!

The former. PipelineDB performs aggregations in memory on microbatches of events, and only merges the aggregate output of each microbatch with what's on disk. This is really the core idea behind why PipelineDB is so performant for continuous time-series aggregation. Microbatch size is configurable: http://docs.pipelinedb.com/conf.html.

Re: PipelineDB 1.0 – High-Performance Time-Series Aggregation for PostgreSQL

#30

Post by alakin »

grammr wrote: post #29
alakin wrote: post #28

Is most of the intermediate processing done in memory, or is it limited by hd write speed?

I'm Derek, one of the co-founders--excellent question! The former. PipelineDB performs aggregations in memory on microbatches of events, and only merges the aggregate output of each microbatch with what's on disk. This is really the core idea behind why PipelineDB is so performant for continuous time-series aggregation. Microbatch size is configurable: http://docs.pipelinedb.com/conf.html .

That's awesome! If you don't mind - one more q.. I see that stream-stream joins are not yet supported (http://docs.pipelinedb.com/joins.html#stream-stream-joins). Can you comment on when you think this feature cold land or is it still a ways off?
Post reply on HN