31–40 of 54 posts
Esper does exactly this - you run streams of events over it and it continuously executes SQL to see if it matches. If so you can:
- run code
- make new streams
- store the results
Esper's been doing this kind of thing for 9 years now.
This looks very cool. Although, I'm not sure I totally understand how it can be used to replace batch ETL processes. So, PipelineDB eliminates ETL batch processing by incrementally inserting data into continuous views, but the documentation says that it's not meant for ad-hoc data warehouses as the raw data is discarded. So, does that leave me still using batch processes to load my data warehouse? Is PipelineDB going…
In terms of not requiring that raw data be stored, a typical setup is to keep raw data somewhere cheap (like S3) so that it's there when you need it. But granular data is often overwhelmingly cold and never looked at again so it may not always be necessary to store it all in an interactively queryable datastore.
As I mentioned, PipelineDB certainly doesn't aim to be a monolithic replacement for all adjacent data processing technologies, but there are areas where it can definitely introduce significant efficiency.
I downloaded the OSX .pkg installer and didn't see anything in /Applications or /opt after running it and telling it to install to my root drive. Just glancing at some docs on your site I see pipeline-init, so doing a find on / to find out where it placed the binaries see it installed to: /usr/lib/pipelinedb/usr/lib/pipelinedb/bin/pipeline-init Is this intentional? EDIT: After playing around with the .pkg file it loo…
This looks very cool. Although, I'm not sure I totally understand how it can be used to replace batch ETL processes. So, PipelineDB eliminates ETL batch processing by incrementally inserting data into continuous views, but the documentation says that it's not meant for ad-hoc data warehouses as the raw data is discarded. So, does that leave me still using batch processes to load my data warehouse? Is PipelineDB going…
Hey Chad, PipelineDB co-founder here. PipelineDB certainly isn't intended to be the only tool in your data infrastructure. But whenever the same queries are being repeatedly run on granular data, those are the types of situations in which it often makes a lot sense to just compute the condensed result incrementally with a continuous view, because that's the only lens it's ever viewed through anyways (dashboards are a…
I downloaded the OSX .pkg installer and didn't see anything in /Applications or /opt after running it and telling it to install to my root drive. Just glancing at some docs on your site I see pipeline-init, so doing a find on / to find out where it placed the binaries see it installed to: /usr/lib/pipelinedb/usr/lib/pipelinedb/bin/pipeline-init Is this intentional? EDIT: After playing around with the .pkg file it loo…
Sorry about that, OSX packaging is a bit finicky right now. We'll publish a package with more sensible defaults very shortly.
Earlier quoted context omitted.
There is also TelegraphCQ, a competing project at Berkeley around the same time; TelegraphCQ was also built on top of PostgreSQL and its support for Continuous Queries seems essentially identical to Continuous Views here ("materialized views", "triggers", and "continuous queries" are all quite similar to each other in terms of the underlying technology needed). TelegraphCQ was commercialized as Truviso, bought out by…
Thanks for the link, I hadn't seen TelegraphCQ previously. Following the trail, I also came across a couple other similar research projects relating to Stream-oriented DBs. Specifically, STREAM from Stanford and Cougar from Cornell, though it appears that all of these academic projects are dormant at this point.
Truviso got bought by Cisco and disappeared into their internal projects.
StreamBase got bought by TIBCO and is still available today.
I downloaded the OSX .pkg installer and didn't see anything in /Applications or /opt after running it and telling it to install to my root drive. Just glancing at some docs on your site I see pipeline-init, so doing a find on / to find out where it placed the binaries see it installed to: /usr/lib/pipelinedb/usr/lib/pipelinedb/bin/pipeline-init Is this intentional? EDIT: After playing around with the .pkg file it loo…
Sorry about that, OSX packaging is a bit finicky right now. We'll publish a package with more sensible defaults very shortly.
I downloaded the OSX .pkg installer and didn't see anything in /Applications or /opt after running it and telling it to install to my root drive. Just glancing at some docs on your site I see pipeline-init, so doing a find on / to find out where it placed the binaries see it installed to: /usr/lib/pipelinedb/usr/lib/pipelinedb/bin/pipeline-init Is this intentional? EDIT: After playing around with the .pkg file it loo…
Sorry about that, OSX packaging is a bit finicky right now. We'll publish a package with more sensible defaults very shortly.
Earlier quoted context omitted.
Sorry about that, OSX packaging is a bit finicky right now. We'll publish a package with more sensible defaults very shortly.
Great thanks! Looking forward to checking it out.
Please shoot me an email (I'm Derek) if you have any issues installing this package. Thanks for your patience!