To Be Continuous

#1

Post by Fergi » Tue, Jul 07, 2015, 6:51 PM UTC

Re: To Be Continuous

#2

Post by zallarak » Tue, Jul 07, 2015, 7:19 PM UTC

Very cool that it is open sourced - seems like there would be a lot to learn from the code. Link: https://github.com/pipelinedb/pipelinedb

Re: To Be Continuous

#3

Post by maslam » Tue, Jul 07, 2015, 7:41 PM UTC

Wonderful! Great job.

Re: To Be Continuous

#4

Post by jxramos » Tue, Jul 07, 2015, 8:15 PM UTC

Well said! Good timing too, I'm beginning to sketch out how to tackle this large file set processing that has to stitch together data from corresponding files. The magnitude I'm imagining is such that I can't just read all the files into memory and do the matching, number crunching, and what not against. I like the concepts and terminology in this article. Definitely worth keeping in the back pocket going forward if not diving into it all outright. Thanks so much.

Re: To Be Continuous

#5

Post by [deleted] » Tue, Jul 07, 2015, 8:15 PM UTC

[deleted]

Re: To Be Continuous

#6

Post by [deleted] » Tue, Jul 07, 2015, 8:15 PM UTC

[deleted]

Re: To Be Continuous

#7

Post by WaxProlix » Tue, Jul 07, 2015, 8:18 PM UTC

As someone who's made a lot of use of `tail` and similar, this is appealing.

But I don't have a lot of use cases in personal projects, and am unlikely to find a good use-case at work in the near future. What's the 'adoption path' for something like this?

I think a really robust sample data set with example queries (think the neo4j imdb examples) would be a great way to show how powerful and easy something like this can be.

Re: To Be Continuous

#8

Post by Fergi » Tue, Jul 07, 2015, 8:22 PM UTC

WaxProlix wrote: post #7

As someone who's made a lot of use of `tail` and similar, this is appealing. But I don't have a lot of use cases in personal projects, and am unlikely to find a good use-case at work in the near future. What's the 'adoption path' for something like this? I think a really robust sample data set with example queries (think the neo4j imdb examples) would be a great way to show how powerful and easy something like this c…

Jeff from PipelineDB here. Great suggestion! We've actually created this using sample Wikipedia data. See:

http://docs.pipelinedb.com/quickstart.html

Re: To Be Continuous

#9

Post by mfenniak » Tue, Jul 07, 2015, 8:41 PM UTC

Cool. Very cool.

My first thought (aside from "Cool") was that the current time would be the tricky thing that can't be incorporated into a continuous view. But even that seems to be handled! http://docs.pipelinedb.com/sliding-windows.html

Looks pretty impressive. :-)

Re: To Be Continuous

#10

Post by pradn » Tue, Jul 07, 2015, 9:03 PM UTC

What does ETL mean in this context?