Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pypeline: A Python library for creating concurrent data pipelines

github.com

Post reply on HN
Print view View original

21–30 of 48 posts

Re: Pypeline: A Python library for creating concurrent data pipelines

#21

Post by dec0dedab0de »

dajohnson89 wrote: post #16
anentropic wrote: post #7

Too much abbreviation! pypeline --> pypeln multiprocessing pipeline --> pr threads pipeline --> th asyncio pipeline --> io this is totally unnecessary If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr` Your library shouldn't export them like this as the default. `io` is especially bad since this overshadows the `io` module in the Python stdlib

At risk of speaking from ignorance (i dont really know python), isn't this succinctness an aim of python? A lot of python enthusiasts i speak with decry the verbosity of Java and point to how much less code it takes to do the same thing in python.

It's really more about readability, In this case threads_pipeline is much easier to understand than th. Descriptive names also means needing less comments.

Re: Pypeline: A Python library for creating concurrent data pipelines

#22

Post by memoir2comment »

roel_v wrote: post #12

None of these frameworks (there are many) seem to have support for repeating a certain target multiple times, with different arguments. For example, say you have a data set with per-country data; how do you repeat the same analysis on each country? This simple example is easy with a loop, but when you have multiple dimensions like this, you want to call each target with all possible permutations, depending on which t…

snakemake does this trivially:

    rule analyze_country:
        input: 'whatever.{country}.txt'
        output: 'analysis.{country}.txt'
        shell:
            'run-analysis-on-country {input} {output} --country=country'

    rule analyze_target_countries:
        input: ['analysis.usa.txt', 'analysis.canada.txt', 'analysis.mexico.txt']

Re: Pypeline: A Python library for creating concurrent data pipelines

#23

Post by agumonkey »

trymas wrote: post #18
dajohnson89 wrote: post #16

Earlier quoted context omitted.

At risk of speaking from ignorance (i dont really know python), isn't this succinctness an aim of python? A lot of python enthusiasts i speak with decry the verbosity of Java and point to how much less code it takes to do the same thing in python.

Python has a famous adage `explicit better than implicit`. You want to be explicit but still concise while writing python. I do not have any experience with Java, but I guess when writing Java you can feel that you use too many words than needed (i.e. definition of verbose [0]), wikipedia has a hello world example[1] and it feels just heavy. IMHO if you write pythonic code, very often it feels like writing/reading pr…

I'm guilty of this, because words are so meaningless to me. I'd rather have single lettre or glyph than ~clear yet ambiguous words.

Re: Pypeline: A Python library for creating concurrent data pipelines

#25

Post by neuromantik8086 »

chrisjc wrote: post #15

Seems like a good time to link to this curated list of pipeline toolkits (not all python). https://github.com/pditommaso/awesome-pipeline/blob/master/R...

This too:

https://github.com/common-workflow-language/common-workflow-...

Also, whenever these conversation of flow-based / piplining tools come up, I always like to point people to Common Workflow Language to remind people that there is an attempt at standardizing workflow descriptions so that they can be used with different packages:

https://www.commonwl.org/

Re: Pypeline: A Python library for creating concurrent data pipelines

#26

Post by cgarciae »

anentropic wrote: post #7

Too much abbreviation! pypeline --> pypeln multiprocessing pipeline --> pr threads pipeline --> th asyncio pipeline --> io this is totally unnecessary If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr` Your library shouldn't export them like this as the default. `io` is especially bad since this overshadows the `io` module in the Python stdlib

Point taken! Thanks a lot for your feedback. Just a few points: * pypeline is already taken :( * My main reason for this was because initially I was thinking that you did an `import pypeln as pl` and then called things like e.g. `pl.pr.map` since you cant abbreviate the module inside `pl` then I picked short names, but then I decided to go for and import the module kind of strategy.

I am thinking about expanding the module names to their worker names: * pr --> process * th --> thread * io --> task

And then have the conventions * from pypeln import process as pr * from pypeln import thread as th * from pypeln import task as io # as ta?

This conversation is very valuable, thank you all for the feedback.

Re: Pypeline: A Python library for creating concurrent data pipelines

#28

Post by cgarciae »

marcyb5st wrote: post #4
adamcharnock wrote: post #2

Pypeline was designed to solve simple medium data tasks that require concurrency and parallelism but where using frameworks like Spark or Dask feel exaggerated or unnatural. This is exactly what I was looking for very recently. Thank you for writing this, I'll certainly look into it.

What about Apache Beam? Getting started with the Python SDK has been very easy IMHO. Also, you are future proof as you can easily switch runner from Local to Dataflow/Flink/...

I use BEAM for my Dataflow jobs. But their local "DirectRunner" is just for testing purposes. As with Spark, BEAM is a huge beast, Pypeline was created with simplicity in mind, its a pure python library, no dependencies.

Re: Pypeline: A Python library for creating concurrent data pipelines

#29

Post by neuromantik8086 »

cgarciae wrote: post #26
anentropic wrote: post #7

Too much abbreviation! pypeline --> pypeln multiprocessing pipeline --> pr threads pipeline --> th asyncio pipeline --> io this is totally unnecessary If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr` Your library shouldn't export them like this as the default. `io` is especially bad since this overshadows the `io` module in the Python stdlib

Point taken! Thanks a lot for your feedback. Just a few points: * pypeline is already taken :( * My main reason for this was because initially I was thinking that you did an `import pypeln as pl` and then called things like e.g. `pl.pr.map` since you cant abbreviate the module inside `pl` then I picked short names, but then I decided to go for and import the module kind of strategy. I am thinking about expanding the…

"import pypeln as pl" could cause quite a bit of confusion in Poland I would imagine.

Re: Pypeline: A Python library for creating concurrent data pipelines

#30

Post by cgarciae »

TBastiani wrote: post #8

mpipe might also be of interest. http://vmlaker.github.io/mpipe/

Thanks! Did take a look at mpipe (its actually referenced in the readme). But mpipe has its flaws:

1. It uses None as the stage terminator, this is VERY error prone, what if you actually want to send None? Pypeline uses a special private terminator.

2. You have to first manually put all the data into the pipe in a for-loop and then manually get it out. In Pypeline all this is simplified: it consumes iterables and all stages are iterables, so its 100% compatible with any function/framework that accepts iterables.

Post reply on HN