Pypeline: A Python library for creating concurrent data pipelines
1–10 of 48 posts
Re: Pypeline: A Python library for creating concurrent data pipelines
#2 Pypeline was designed to solve simple medium
data tasks that require concurrency
and parallelism but where using frameworks
like Spark or Dask feel exaggerated or unnatural.
Re: Pypeline: A Python library for creating concurrent data pipelines
#3I wonder if you might compare this to Bonobo [https://www.bonobo-project.org/] which I think has similar design goals?
Re: Pypeline: A Python library for creating concurrent data pipelines
#4Pypeline was designed to solve simple medium data tasks that require concurrency and parallelism but where using frameworks like Spark or Dask feel exaggerated or unnatural. This is exactly what I was looking for very recently. Thank you for writing this, I'll certainly look into it.
What about Apache Beam? Getting started with the Python SDK has been very easy IMHO. Also, you are future proof as you can easily switch runner from Local to Dataflow/Flink/...
Re: Pypeline: A Python library for creating concurrent data pipelines
#5[deleted]
Re: Pypeline: A Python library for creating concurrent data pipelines
#6I wonder if you might compare this to Bonobo [ https://www.bonobo-project.org/ ] which I think has similar design goals?
Pypeline is a library you use in your code while Bonobo seems to be a framework that use your code.
I tend to think that you lose flexibility with the latter.
Re: Pypeline: A Python library for creating concurrent data pipelines
#7Too much abbreviation!
pypeline --> pypeln
multiprocessing pipeline --> pr
threads pipeline --> th
asyncio pipeline --> io
this is totally unnecessary
If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr`
Your library shouldn't export them like this as the default.
`io` is especially bad since this overshadows the `io` module in the Python stdlib
Re: Pypeline: A Python library for creating concurrent data pipelines
#8mpipe might also be of interest.
Re: Pypeline: A Python library for creating concurrent data pipelines
#9Too much abbreviation! pypeline --> pypeln multiprocessing pipeline --> pr threads pipeline --> th asyncio pipeline --> io this is totally unnecessary If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr` Your library shouldn't export them like this as the default. `io` is especially bad since this overshadows the `io` module in the Python stdlib
Absolutely agree with this.
Re: Pypeline: A Python library for creating concurrent data pipelines
#10Snakemake [0] is a tool worth checking out. You can use it to create declarative workflows, and similar to make, it creates a DAG of dependencies when you give it your desired output. Each rule can specify how many threads it needs and other arbitrary resources and the scheduler uses that to constrain execution. Workflows are architecture independent - you should be able to execute a snakemake workflow on a laptop, in the cloud, or on an HPC cluster.
It also allows you to use UNIX pipes with your dependent jobs when that is appropriate [1].
[0] https://snakemake.readthedocs.io/en/stable/index.html
[1] https://snakemake.readthedocs.io/en/stable/snakefiles/rules....