Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To Be Continuous

pipelinedb.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: To Be Continuous

#41

Post by ak4g »

perplexes wrote: post #31

I'm surprised no one has mentioned Esper yet: http://www.espertech.com/esper/ Esper does exactly this - you run streams of events over it and it continuously executes SQL to see if it matches. If so you can: - run code - make new streams - store the results Esper's been doing this kind of thing for 9 years now.

Esper is really curious. I looked at it while getting a feel for the general landscape of streaming data tools in late 2013 - to be clear, it was a very cursory look, on the timescale of a few hours at most. I quote what my final take-away was at the time:

> Seems good but... such a weird project. Codehaus, svn, not high activity, but consistent, stable releases for five years. Maybe just not the kind of thing webdevs get into? Not sure if that's a strike against or not.

With the demise of Codehaus it looks like they've moved to github:

https://github.com/espertechinc/esper

But oddly they don't seem to have migrated their svn history, and the README implies they don't plan to... I certainly hope they didn't lose it when Codehaus shut down. There was, as noted, already 9 years worth of code changes in that repo. That would be unfortunate.

Re: To Be Continuous

#42

Post by ak4g »

jchrisa wrote: post #23
infodroid wrote: post #15

It looks like PipelineDB is implemented as a fork of PostgreSQL. I would be interested to understand what is different about the architecture of PipelineDB that it couldn't be integrated into upstream PostgreSQL.

Cofounder at Couchbase here. Is the license decision driven by business or is there some dependency that pushes you to GPL? For us Apache 2.0 has been worth it even when other companies use our code in their products.

And not just the GPL, but the AGPL. That one caught me by surprise. Does this mean there is going to be a MySQL-style dual-licensing situation in the future? (or is this already the case?)

Re: To Be Continuous

#43

Post by ScottBurson »

assface wrote: post #37
tuckermi wrote: post #25

Earlier quoted context omitted.

Thanks for the link, I hadn't seen TelegraphCQ previously. Following the trail, I also came across a couple other similar research projects relating to Stream-oriented DBs. Specifically, STREAM from Stanford and Cougar from Cornell, though it appears that all of these academic projects are dormant at this point.

All of the academic stream systems from the early 2000s are long over. The students have all graduated. Truviso got bought by Cisco and disappeared into their internal projects. StreamBase got bought by TIBCO and is still available today.

And Coral8 got bought by Sybase and became Sybase CEP, also still available. (Sybase is now owned by SAP.)

Re: To Be Continuous

#44

Post by Fergi »

ak4g wrote: post #42
jchrisa wrote: post #23

Earlier quoted context omitted.

Cofounder at Couchbase here. Is the license decision driven by business or is there some dependency that pushes you to GPL? For us Apache 2.0 has been worth it even when other companies use our code in their products.

And not just the GPL, but the AGPL. That one caught me by surprise. Does this mean there is going to be a MySQL-style dual-licensing situation in the future? (or is this already the case?)

PipelineDB is actually licensed under the GPLv3. We accidentally updated our files with the AGPL earlier today by mistake but have since rectified that. Apologies for the mixup!

Re: To Be Continuous

#45

Post by asgard1024 »

This claim about ETL not needed in the future sounds dubious. I work on a large application that is all about ETL. If we wanted to use this new method instead, I am not sure how it would deal with the following:

- State in the data. In many sources we have, processing depends on some internal state, which must be kept along the time. For example some process has started and we will know when it ended, and we must keep its state so we could correctly process the ending event (to match it up). I am not clear how this will work with continuous views. I would say this is actually the major reason of what makes ETL processing non-trivial.

- Processing failure. Let's say something goes wrong and the data processing fails (or it can actually be even planned downtime). How do we know where to restart, to avoid processing data twice or miss data? Does the continuous stream take care of this metadata? And how does it deal with the state information per above? If you do data processing in batches, there is an obvious point of restart. Again, I think the extra complexity that "continuous" approach says is unnecessary relates to the fact that you want to be able to checkpoint the state of processing for various reasons.

Re: To Be Continuous

#46

Post by reitanqild »

Fergi wrote: post #44
ak4g wrote: post #42

Earlier quoted context omitted.

And not just the GPL, but the AGPL. That one caught me by surprise. Does this mean there is going to be a MySQL-style dual-licensing situation in the future? (or is this already the case?)

PipelineDB is actually licensed under the GPLv3. We accidentally updated our files with the AGPL earlier today by mistake but have since rectified that. Apologies for the mixup!

As much as thank you comments are downvoted here I'll still say: Thank you! Same goes to everyone else who avoids hasseling developers everywhere with AGPL.

Re: To Be Continuous

#47

Post by sheraz »

Can PipelineDB be used to run projections for an EventStore?

I'm experimenting with the EventStore pattern for a side project, and I have struggled to implement projections. Could PipelineDB be a way to deliver that?

Re: To Be Continuous

#48

Post by anarazel »

buremba wrote: post #34

We needed to implement continuous queries in our application code. (It's actually hard to do it right in Postgresql so it's very limited) https://github.com/buremba/rakam/wiki/Postgresql-Backend#con... Since stream processing and real-time analytics are quite hot topics nowadays, I think real-time databases will get much more attention in a near future.

Postgres actually does expose a commit log mechanism since 9.4, expanded with 9.5.

http://www.postgresql.org/docs/devel/static/logicaldecoding.... and 9.5's track_commit_timestamp = on.

Re: To Be Continuous

#49

Post by reubano »

grammr wrote: post #32
chadthenderson wrote: post #27

This looks very cool. Although, I'm not sure I totally understand how it can be used to replace batch ETL processes. So, PipelineDB eliminates ETL batch processing by incrementally inserting data into continuous views, but the documentation says that it's not meant for ad-hoc data warehouses as the raw data is discarded. So, does that leave me still using batch processes to load my data warehouse? Is PipelineDB going…

Hey Chad, PipelineDB co-founder here. PipelineDB certainly isn't intended to be the only tool in your data infrastructure. But whenever the same queries are being repeatedly run on granular data, those are the types of situations in which it often makes a lot sense to just compute the condensed result incrementally with a continuous view, because that's the only lens it's ever viewed through anyways (dashboards are a…

So what's the best practice for when you want a real time dashboard but also want the ability to compare data overtime. E.g., ave. bounce rate this month vs last? Is Pipeline still ideal in this case?

Re: To Be Continuous

#50

Post by cuttinedg »

grammr wrote: post #40
res0nat0r wrote: post #38

Earlier quoted context omitted.

Great thanks! Looking forward to checking it out.

We just updated http://www.pipelinedb.com/download/0.7.7/osx . Please shoot me an email (I'm Derek) if you have any issues installing this package. Thanks for your patience!

There was an error with init and created an issue in github

https://github.com/pipelinedb/pipelinedb/issues/1025

Post reply on HN