mpipe might also be of interest. http://vmlaker.github.io/mpipe/
3. If you first have to put in all the data first you cant handle streams. Pypeline accepts non-terminating iterables and also gives you back possibly non-terminating iterables.
Pypeline: A Python library for creating concurrent data pipelines
31–40 of 48 posts
Re: Pypeline: A Python library for creating concurrent data pipelines
#32Looks similar to what tf.data does for Tensorflow
Nice to hear that. When I wrote this:
"Pypeline was designed to solve simple medium data tasks that require concurrency and parallelism but where using frameworks like Spark or Dask feel exaggerated or unnatural."
it was actually because I've resorted / hacked into tf.data and Dask in the past just to get concurrency and parallelism. Pypeline is way more natural for pure python stuff.
Re: Pypeline: A Python library for creating concurrent data pipelines
#33Earlier quoted context omitted.
Point taken! Thanks a lot for your feedback. Just a few points: * pypeline is already taken :( * My main reason for this was because initially I was thinking that you did an `import pypeln as pl` and then called things like e.g. `pl.pr.map` since you cant abbreviate the module inside `pl` then I picked short names, but then I decided to go for and import the module kind of strategy. I am thinking about expanding the…
"import pypeln as pl" could cause quite a bit of confusion in Poland I would imagine.
And maybe perl...
Re: Pypeline: A Python library for creating concurrent data pipelines
#34None of these frameworks (there are many) seem to have support for repeating a certain target multiple times, with different arguments. For example, say you have a data set with per-country data; how do you repeat the same analysis on each country? This simple example is easy with a loop, but when you have multiple dimensions like this, you want to call each target with all possible permutations, depending on which t…
snakemake does this trivially: rule analyze_country: input: 'whatever.{country}.txt' output: 'analysis.{country}.txt' shell: 'run-analysis-on-country {input} {output} --country=country' rule analyze_target_countries: input: ['analysis.usa.txt', 'analysis.canada.txt', 'analysis.mexico.txt']
Small change, you have to use wilcards.country inside the shell call:
rule analyze_country:
input: 'whatever.{country}.txt'
output: 'analysis.{country}.txt'
shell:
'run-analysis-on-country {input} {output} --country={wildcards.country}'Re: Pypeline: A Python library for creating concurrent data pipelines
#35Earlier quoted context omitted.
Point taken! Thanks a lot for your feedback. Just a few points: * pypeline is already taken :( * My main reason for this was because initially I was thinking that you did an `import pypeln as pl` and then called things like e.g. `pl.pr.map` since you cant abbreviate the module inside `pl` then I picked short names, but then I decided to go for and import the module kind of strategy. I am thinking about expanding the…
"import pypeln as pl" could cause quite a bit of confusion in Poland I would imagine.
jajajajaja
Re: Pypeline: A Python library for creating concurrent data pipelines
#36Too much abbreviation! pypeline --> pypeln multiprocessing pipeline --> pr threads pipeline --> th asyncio pipeline --> io this is totally unnecessary If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr` Your library shouldn't export them like this as the default. `io` is especially bad since this overshadows the `io` module in the Python stdlib
Not a fan of this. You save a couple of chars and massively decrease legibility. Anyone new to the project will find it way harder to understand. Instead of abbreviation hell, just spell it out and move on. Not worth saving a couple of characters.
True masters of naming make sure to use variable names that are long but unambiguous after the first several characters, so that with tab-completion you get the best of both worlds.
Re: Pypeline: A Python library for creating concurrent data pipelines
#37Dask is relatively lightweight actually, because it is pure Python. Also, there is "Streamz" which solves a similar problem, seems more mature and can work with or without Dask or Dask-Distributed.
Dask might be lightweight internally but resorting to it just to solve a simple task that requires concurrency is not "simple".
Streamz looks nice! However:
"Streamz relies on the Tornado framework for concurrency. This allows us to handle many concurrent operations cheaply and consistently within a SINGLE THREAD."
Apparently you can set it up to use Dask to escape the single threads but that is kind of a global config. With Pypeline you can mix and match between using Processes, Threads, and asyncio.Tasks where it makes sense, resource management per stage is simple and explicit. If you have some understanding of the multiprocessing, threading and asyncio modules, Pypeline will save you tons of time.
Still, will keep an eye on Streamz, its a very nice work, lots of features, it should get more visibility.
Re: Pypeline: A Python library for creating concurrent data pipelines
#38Too much abbreviation! pypeline --> pypeln multiprocessing pipeline --> pr threads pipeline --> th asyncio pipeline --> io this is totally unnecessary If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr` Your library shouldn't export them like this as the default. `io` is especially bad since this overshadows the `io` module in the Python stdlib
Point taken! Thanks a lot for your feedback. Just a few points: * pypeline is already taken :( * My main reason for this was because initially I was thinking that you did an `import pypeln as pl` and then called things like e.g. `pl.pr.map` since you cant abbreviate the module inside `pl` then I picked short names, but then I decided to go for and import the module kind of strategy. I am thinking about expanding the…
First of all, it's super cool. :)
There are a lot of "pipe" projects in PyPi, but your project is also about process management. Maybe you should avoid "pipe" in your name perhaps? FlowProcessor? nFlow? xFlow?
I do agree that you should avoid io for asyncio. You should probably at least use aio, but there's no reason you can't have asyncio_task, thread_task, multiprocessing_task.
Lastly, in my mind the killer app for this would be to allow something that works on top of Celery in production, but then be able to fall back to say multiprocessing or threading when running locally. That would allow me to prototype something, and then when I want to scale, I can just change a config setting.
Re: Pypeline: A Python library for creating concurrent data pipelines
#39None of these frameworks (there are many) seem to have support for repeating a certain target multiple times, with different arguments. For example, say you have a data set with per-country data; how do you repeat the same analysis on each country? This simple example is easy with a loop, but when you have multiple dimensions like this, you want to call each target with all possible permutations, depending on which t…
Sounds like a group_by and then do a function per group? Pypeline doesn't have grouping but sounds like Spark or Dask should the the job.
Re: Pypeline: A Python library for creating concurrent data pipelines
#40Earlier quoted context omitted.
Point taken! Thanks a lot for your feedback. Just a few points: * pypeline is already taken :( * My main reason for this was because initially I was thinking that you did an `import pypeln as pl` and then called things like e.g. `pl.pr.map` since you cant abbreviate the module inside `pl` then I picked short names, but then I decided to go for and import the module kind of strategy. I am thinking about expanding the…
First of all, it's super cool. :) There are a lot of "pipe" projects in PyPi, but your project is also about process management. Maybe you should avoid "pipe" in your name perhaps? FlowProcessor? nFlow? xFlow? I do agree that you should avoid io for asyncio. You should probably at least use aio, but there's no reason you can't have asyncio_task, thread_task, multiprocessing_task. Lastly, in my mind the killer app for…
Hey, thanks for all the feedback. I will change the naming since its something most of you have agreed is a good change.
The goal I have for Pypeline is much simpler: let you easily setup data pipelines where you leverage processes, threads and asyncio where they are good at. So in my mind a killer app would be a pipeline that maybe starts with an asyncio stage for e.g. downloading images, maybe then a multiprocess stage for e.g. doing image processing, and finally a threading stage for e.g. interacting with the OS.
Right now I see Pypeline more as an easy to use single machine tool instead of a higher level distributed abstraction like Celery. Maybe other framework could leverage Pypeline to ease their work.