Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Pyper – Concurrent Python Made Simple

github.com

Post reply on HN
Print view View original

11–20 of 38 posts

Re: Show HN: Pyper – Concurrent Python Made Simple

#11

Post by t43562 »

rtpg wrote: post #5

You really should dive more into the `multiprocess` support option and highlight how this gets around issues with the GIL. This feels like a major value add, and "does this help with CPU-bound work" being "yes" is a big deal! I don't really need pipelining that much, but pipelining along with a certain level of durability and easy multiprocessing support? Now we're talking

...although python 3.13 can be built without the GIL and it really does make threading useful. I did some comparisons with and without.

I suppose one excellent thing about this would be if you could just change 1 parameter and switch from multiprocessing to threaded.

Re: Show HN: Pyper – Concurrent Python Made Simple

#12

Post by rtpg »

t43562 wrote: post #11
rtpg wrote: post #5

You really should dive more into the `multiprocess` support option and highlight how this gets around issues with the GIL. This feels like a major value add, and "does this help with CPU-bound work" being "yes" is a big deal! I don't really need pipelining that much, but pipelining along with a certain level of durability and easy multiprocessing support? Now we're talking

...although python 3.13 can be built without the GIL and it really does make threading useful. I did some comparisons with and without. I suppose one excellent thing about this would be if you could just change 1 parameter and switch from multiprocessing to threaded.

I think you could build off of threading. I do think here it's good to acknowledge that Python async is fundamentally single-threaded (or rather, "single thread per event loop"), so if you do go a multi-thread version you might have to do some bookkeeping to make it all work well.

I'm not sure how well async Python libs are tested against working in a world with multiple event loops, but I bet there are a _lot_ of latent bugs in that space.

Re: Show HN: Pyper – Concurrent Python Made Simple

#14

Post by giancarlostoro »

Lowkey I hate the "\" line continuation in Python to force PEP-8 compliance in a way... Is there any Pythonistas who would write the examples in there differently to achieve a similar level of readability?

> pipeline = task(get_data, branch=True) \

> | task(step1, workers=20) \

> | task(step2, workers=20) \

> | task(step3, workers=20, multiprocess=True)

Re: Show HN: Pyper – Concurrent Python Made Simple

#15

Post by morkalork »

giancarlostoro wrote: post #14

Lowkey I hate the "\" line continuation in Python to force PEP-8 compliance in a way... Is there any Pythonistas who would write the examples in there differently to achieve a similar level of readability? > pipeline = task(get_data, branch=True) \ > | task(step1, workers=20) \ > | task(step2, workers=20) \ > | task(step3, workers=20, multiprocess=True)

Wrap the statement in (...) and you can drop the backslashes. See also how people split up complicated loc queries in pandas on multiple lines too.

Re: Show HN: Pyper – Concurrent Python Made Simple

#16

Post by giancarlostoro »

morkalork wrote: post #15
giancarlostoro wrote: post #14

Lowkey I hate the "\" line continuation in Python to force PEP-8 compliance in a way... Is there any Pythonistas who would write the examples in there differently to achieve a similar level of readability? > pipeline = task(get_data, branch=True) \ > | task(step1, workers=20) \ > | task(step2, workers=20) \ > | task(step3, workers=20, multiprocess=True)

Wrap the statement in (...) and you can drop the backslashes. See also how people split up complicated loc queries in pandas on multiple lines too.

Thank you! It seems this works for not just parens but square brackets and curly braces too! Only special requirement is indentation is consistent (duh though).

I've not been doing Python day-to-day so I'm starting to lose my touch on all the nice little tricks.

Re: Show HN: Pyper – Concurrent Python Made Simple

#17

Post by Rickster35 »

giancarlostoro wrote: post #16
morkalork wrote: post #15

Earlier quoted context omitted.

Wrap the statement in (...) and you can drop the backslashes. See also how people split up complicated loc queries in pandas on multiple lines too.

Thank you! It seems this works for not just parens but square brackets and curly braces too! Only special requirement is indentation is consistent (duh though). I've not been doing Python day-to-day so I'm starting to lose my touch on all the nice little tricks.

Yeah, this seems neatest if you don't like line breaks

pipeline = (

    task(get_data, branch=True)

    | task(step1, workers=20)

    | task(step2, workers=20)

    | task(step3, workers=20, multiprocess=True)
)

Square brackets would create a list and braces would create a set of course. The contents still can be split over different lines-- just pointing that this syntax doesn't do the same thing.

Re: Show HN: Pyper – Concurrent Python Made Simple

#18

Post by giancarlostoro »

Rickster35 wrote: post #17
giancarlostoro wrote: post #16

Earlier quoted context omitted.

Thank you! It seems this works for not just parens but square brackets and curly braces too! Only special requirement is indentation is consistent (duh though). I've not been doing Python day-to-day so I'm starting to lose my touch on all the nice little tricks.

Yeah, this seems neatest if you don't like line breaks pipeline = ( task(get_data, branch=True) | task(step1, workers=20) | task(step2, workers=20) | task(step3, workers=20, multiprocess=True) ) Square brackets would create a list and braces would create a set of course. The contents still can be split over different lines-- just pointing that this syntax doesn't do the same thing.

Yeah I should have noted that they behave the same in terms of cleaning up a line, not necessarily in terms of what the compiler will do with each token pair.

Re: Show HN: Pyper – Concurrent Python Made Simple

#19

Post by cess11 »

globular-toast wrote: post #10
solidasparagus wrote: post #4

Nice work! There is a gap when it comes to writing single-machine, concurrent CPU-bound python code. Ray is too big, pykka is threads only, builtins are poorly abstracted. The syntax is also very nice! But I'm not sure I can use this even though I have a specific use-case that feels like it would work well (high-performance pure Python downloading from cloud object storage). The examples are a bit too simple and I do…

Do you really need to reinvent the wheel every time for parallel workloads? Just learn GNU parallel and write single-threaded code. Concurrency in general isn't about parallelism. It's just about doing multiple things at the same time.

GNU Parallel is really neat, software that's so good it's boring. Closing in on being a quarter century old by now, no? I remember first reading about it in 2003 maybe?

I've also used 'fork in Picolisp a lot for this kind of thing, and also Elixir, which arguably has much nicer pipes.

But hey, it's good that Python after like thirty years or so is trying to get decent concurrency. Eventually people that use it as a first language might learn about such things too.

Re: Show HN: Pyper – Concurrent Python Made Simple

#20

Post by asplake »

Rickster35 wrote: post #17
giancarlostoro wrote: post #16

Earlier quoted context omitted.

Thank you! It seems this works for not just parens but square brackets and curly braces too! Only special requirement is indentation is consistent (duh though). I've not been doing Python day-to-day so I'm starting to lose my touch on all the nice little tricks.

Yeah, this seems neatest if you don't like line breaks pipeline = ( task(get_data, branch=True) | task(step1, workers=20) | task(step2, workers=20) | task(step3, workers=20, multiprocess=True) ) Square brackets would create a list and braces would create a set of course. The contents still can be split over different lines-- just pointing that this syntax doesn't do the same thing.

The example at https://pyper-dev.github.io/pyper/docs/Examples/ChessDataAna... does exactly that.
Post reply on HN