Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pypeline: A Python library for creating concurrent data pipelines

github.com

Post reply on HN
Print view View original

11–20 of 48 posts

Re: Pypeline: A Python library for creating concurrent data pipelines

#11

Post by philote »

marcyb5st wrote: post #4
adamcharnock wrote: post #2

Pypeline was designed to solve simple medium data tasks that require concurrency and parallelism but where using frameworks like Spark or Dask feel exaggerated or unnatural. This is exactly what I was looking for very recently. Thank you for writing this, I'll certainly look into it.

What about Apache Beam? Getting started with the Python SDK has been very easy IMHO. Also, you are future proof as you can easily switch runner from Local to Dataflow/Flink/...

But not future proof in having to use python 2.7 unfortunately.

Re: Pypeline: A Python library for creating concurrent data pipelines

#12

Post by roel_v »

None of these frameworks (there are many) seem to have support for repeating a certain target multiple times, with different arguments. For example, say you have a data set with per-country data; how do you repeat the same analysis on each country? This simple example is easy with a loop, but when you have multiple dimensions like this, you want to call each target with all possible permutations, depending on which type of dimension is actually relevant for that target. Does any ETL framework support that?

(I was actually just writing a spec for a new tool that does just this this afternoon because I can't find anything suitable)

Re: Pypeline: A Python library for creating concurrent data pipelines

#13

Post by ricw »

anentropic wrote: post #7

Too much abbreviation! pypeline --> pypeln multiprocessing pipeline --> pr threads pipeline --> th asyncio pipeline --> io this is totally unnecessary If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr` Your library shouldn't export them like this as the default. `io` is especially bad since this overshadows the `io` module in the Python stdlib

Not a fan of this. You save a couple of chars and massively decrease legibility. Anyone new to the project will find it way harder to understand. Instead of abbreviation hell, just spell it out and move on. Not worth saving a couple of characters.

Re: Pypeline: A Python library for creating concurrent data pipelines

#16

Post by dajohnson89 »

anentropic wrote: post #7

Too much abbreviation! pypeline --> pypeln multiprocessing pipeline --> pr threads pipeline --> th asyncio pipeline --> io this is totally unnecessary If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr` Your library shouldn't export them like this as the default. `io` is especially bad since this overshadows the `io` module in the Python stdlib

At risk of speaking from ignorance (i dont really know python), isn't this succinctness an aim of python? A lot of python enthusiasts i speak with decry the verbosity of Java and point to how much less code it takes to do the same thing in python.

Re: Pypeline: A Python library for creating concurrent data pipelines

#17

Post by sebcat »

anentropic wrote: post #7

Too much abbreviation! pypeline --> pypeln multiprocessing pipeline --> pr threads pipeline --> th asyncio pipeline --> io this is totally unnecessary If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr` Your library shouldn't export them like this as the default. `io` is especially bad since this overshadows the `io` module in the Python stdlib

Agreed. When it comes to naming things, I really like the way Russ Cox puts it: https://research.swtch.com/names

Re: Pypeline: A Python library for creating concurrent data pipelines

#18

Post by trymas »

dajohnson89 wrote: post #16
anentropic wrote: post #7

Too much abbreviation! pypeline --> pypeln multiprocessing pipeline --> pr threads pipeline --> th asyncio pipeline --> io this is totally unnecessary If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr` Your library shouldn't export them like this as the default. `io` is especially bad since this overshadows the `io` module in the Python stdlib

At risk of speaking from ignorance (i dont really know python), isn't this succinctness an aim of python? A lot of python enthusiasts i speak with decry the verbosity of Java and point to how much less code it takes to do the same thing in python.

Python has a famous adage `explicit better than implicit`.

You want to be explicit but still concise while writing python.

I do not have any experience with Java, but I guess when writing Java you can feel that you use too many words than needed (i.e. definition of verbose [0]), wikipedia has a hello world example[1] and it feels just heavy.

IMHO if you write pythonic code, very often it feels like writing/reading prose, seems like talking to computer. You are not too implicit, neither too verbose.

[0] https://www.merriam-webster.com/dictionary/verbose

[1] https://en.wikipedia.org/wiki/Java_(programming_language)#"H...

Re: Pypeline: A Python library for creating concurrent data pipelines

#20

Post by rcthompson »

anentropic wrote: post #7

Too much abbreviation! pypeline --> pypeln multiprocessing pipeline --> pr threads pipeline --> th asyncio pipeline --> io this is totally unnecessary If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr` Your library shouldn't export them like this as the default. `io` is especially bad since this overshadows the `io` module in the Python stdlib

And it's especially unnecessary since Python already supports abbreviation during import. It could just as easily be "from pypeline import process as pr" or something similar.
Post reply on HN