Pypeline: A Python library for creating concurrent data pipelines • Hacker News Board

Re: Pypeline: A Python library for creating concurrent data pipelines

#41

Post by snidane » Mon, Sep 24, 2018, 7:50 PM UTC

From my experience building similar pipelining and reverse polish function application tooling in python.

Piping using the | operator can make tracebacks pretty ugly with some operators.

If you want to keep the code still somewhat 'pythonic' without introducing the syntax magic using |, you can do it similarly:

  range(10)
  | pp.flatmap(lambda x: [x + 1, x + 2])
  | pp.map(lambda x: x * x)
  ...

You can do this instead:

  xs = range(10)
  xs = pp.flatmap(xs, lambda x: [x + 1, x + 2])
  xs = pp.map(xs, lambda x: x * x)
  ...

It helps to keep the operand as first argument, instead of last, because those lambdas are best kept at the end.

So instead of

  map(fn, xs)

do

  map(xs, fn)

Re: Pypeline: A Python library for creating concurrent data pipelines

#42

Post by anentropic » Mon, Sep 24, 2018, 9:38 PM UTC

dajohnson89 wrote: post #16

anentropic wrote: post #7

Too much abbreviation! pypeline --> pypeln multiprocessing pipeline --> pr threads pipeline --> th asyncio pipeline --> io this is totally unnecessary If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr` Your library shouldn't export them like this as the default. `io` is especially bad since this overshadows the `io` module in the Python stdlib

At risk of speaking from ignorance (i dont really know python), isn't this succinctness an aim of python? A lot of python enthusiasts i speak with decry the verbosity of Java and point to how much less code it takes to do the same thing in python.

The succinctness of Python (such as there is) comes from the syntax itself. On the other hand, obscure variable names are discouraged. It's not a 'code golf' language. The aim is clarity and readability - unobscured by either verbose boilerplate or excessive brevity.

A library author should provide names which are descriptive and clear. Users can then abbreviate them however much or little as we choose (by import aliasing). For example it is very common to see `import numpy as np`... but you wouldn't want them to publish the library as `np`. It should have its proper name.

One reason for this is if I'm exploring code in a REPL or IDE with tab-completion. You want to have some idea what a module is for, without having to play 'guess the abbreviation'.

Re: Pypeline: A Python library for creating concurrent data pipelines

#43

Post by anentropic » Mon, Sep 24, 2018, 9:53 PM UTC

cgarciae wrote: post #26

anentropic wrote: post #7

Too much abbreviation! pypeline --> pypeln multiprocessing pipeline --> pr threads pipeline --> th asyncio pipeline --> io this is totally unnecessary If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr` Your library shouldn't export them like this as the default. `io` is especially bad since this overshadows the `io` module in the Python stdlib

Point taken! Thanks a lot for your feedback. Just a few points: * pypeline is already taken :( * My main reason for this was because initially I was thinking that you did an `import pypeln as pl` and then called things like e.g. `pl.pr.map` since you cant abbreviate the module inside `pl` then I picked short names, but then I decided to go for and import the module kind of strategy. I am thinking about expanding the…

Hi, just coming back to say congrats on your library and I wish you all the best with it :)

I see your reasoning here (`import pypeln as pl`) but I still think where you have submodules you should use unabbreviated words for their names.

For me I'd be happy with `pl.process.map` in my code, but `pl.pr.map` feels a bit too obscure to have as the default.

These things are quite subjective of course, but part of that subjective judgement comes from the experience of what is commonly done in other Python libraries (the stdlib is a bit of a mixed bag in this regard unfortunately, riddled with CamelCase and other abominations).

Re: Pypeline: A Python library for creating concurrent data pipelines

#44

Post by bayesian_horse » Tue, Sep 25, 2018, 4:38 AM UTC

cgarciae wrote: post #37

bayesian_horse wrote: post #27

Dask is relatively lightweight actually, because it is pure Python. Also, there is "Streamz" which solves a similar problem, seems more mature and can work with or without Dask or Dask-Distributed.

Dask might be lightweight internally but resorting to it just to solve a simple task that requires concurrency is not "simple". Streamz looks nice! However: "Streamz relies on the Tornado framework for concurrency. This allows us to handle many concurrent operations cheaply and consistently within a SINGLE THREAD." Apparently you can set it up to use Dask to escape the single threads but that is kind of a global conf…

Pypeline doesn't seem that simple itself.

Re: Pypeline: A Python library for creating concurrent data pipelines

#45

Post by PurpleRamen » Tue, Sep 25, 2018, 8:16 AM UTC

cgarciae wrote: post #26

anentropic wrote: post #7

Too much abbreviation! pypeline --> pypeln multiprocessing pipeline --> pr threads pipeline --> th asyncio pipeline --> io this is totally unnecessary If I want to use short abbreviated names in my code I can always `from pypeline import multiprocess_pipeline as pr` Your library shouldn't export them like this as the default. `io` is especially bad since this overshadows the `io` module in the Python stdlib

Point taken! Thanks a lot for your feedback. Just a few points: * pypeline is already taken :( * My main reason for this was because initially I was thinking that you did an `import pypeln as pl` and then called things like e.g. `pl.pr.map` since you cant abbreviate the module inside `pl` then I picked short names, but then I decided to go for and import the module kind of strategy. I am thinking about expanding the…

> My main reason for this was because initially I was thinking that you did an `import pypeln as pl` and then called things like e.g. `pl.pr.map` since you cant abbreviate the module inside `pl` then I picked short names

"Assumptions are the root of all evil."

With autocomplete a coder has no reason to use shortnames anyway.

Re: Pypeline: A Python library for creating concurrent data pipelines

#46

Post by unethical_ban » Tue, Sep 25, 2018, 7:53 PM UTC

bayesian_horse wrote: post #44

cgarciae wrote: post #37

Earlier quoted context omitted.

Dask might be lightweight internally but resorting to it just to solve a simple task that requires concurrency is not "simple". Streamz looks nice! However: "Streamz relies on the Tornado framework for concurrency. This allows us to handle many concurrent operations cheaply and consistently within a SINGLE THREAD." Apparently you can set it up to use Dask to escape the single threads but that is kind of a global conf…

Pypeline doesn't seem that simple itself.

Why not? It seems to be a boilerplate remover for simple parallel processing tasks.

Re: Pypeline: A Python library for creating concurrent data pipelines

#47

Post by bayesian_horse » Wed, Sep 26, 2018, 9:10 AM UTC

unethical_ban wrote: post #46

bayesian_horse wrote: post #44

Earlier quoted context omitted.

Pypeline doesn't seem that simple itself.

Why not? It seems to be a boilerplate remover for simple parallel processing tasks.

Maybe I'm just not quite able to get why "lightweight" is a thing. I also prefer Django over Flask for even the simplest of server software...

Re: Pypeline: A Python library for creating concurrent data pipelines

#48

Post by bb88 » Sun, Sep 30, 2018, 1:02 AM UTC

cgarciae wrote: post #40

bb88 wrote: post #38

Earlier quoted context omitted.

First of all, it's super cool. :) There are a lot of "pipe" projects in PyPi, but your project is also about process management. Maybe you should avoid "pipe" in your name perhaps? FlowProcessor? nFlow? xFlow? I do agree that you should avoid io for asyncio. You should probably at least use aio, but there's no reason you can't have asyncio_task, thread_task, multiprocessing_task. Lastly, in my mind the killer app for…

Hey, thanks for all the feedback. I will change the naming since its something most of you have agreed is a good change. The goal I have for Pypeline is much simpler: let you easily setup data pipelines where you leverage processes, threads and asyncio where they are good at. So in my mind a killer app would be a pipeline that maybe starts with an asyncio stage for e.g. downloading images, maybe then a multiprocess s…

So then I would want to have one more stage... a celery stage for when you want to cluster work across multiple machines. :)