Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Pyper – Concurrent Python Made Simple

github.com

Post reply on HN
Print view View original

21–30 of 38 posts

Re: Show HN: Pyper – Concurrent Python Made Simple

#21

Post by JackC »

Cool, I was just looking for something like this!

It's surprisingly annoying in built-in python to do something like this. The most recent thing I was trying to do was:

- load URLs from a file - hand them out to one subprocess per cpu - download them concurrently in threads or async within each subprocess - pull the results back into a single process for formatting and storing

Getting this to work and handle queues, ctrl-c, exceptions etc. is just a whole mess involving python builtins created at different times with different interfaces; I hacked until I kind of got it working, but didn't love it. Bundling it all in a single tested package would be great.

Re: Show HN: Pyper – Concurrent Python Made Simple

#24

Post by gpderetta »

From just a short skimming of the docs:

- my biggest issue with concurrency in python (especially with asyncio) is leaking tasks. Pyper should provide structured concurrency support a-la trio.

- I don't see the opposite of branch to collect the output of multiple sub pipelines into a single stage. I need this pretty much always and it is a chore to implement.

- Async need not force the full pipeline to be async. There should be an option to run async funcitons in background event loops. Especially as you already support threaded executions.

Re: Show HN: Pyper – Concurrent Python Made Simple

#25

Post by ge96 »

gpderetta wrote: post #23
ge96 wrote: post #22

I still gotta wrap my head around concurrency, I'm not sure if using threads count for concurrency

of course it does, why wouldn't they?

Something about GIL and how code is written (one statement comes before the other), true parallelism, anyway I gotta do my hw

I have not done anything significant like HFT to really dig deep into this

Also coming from JS async/await (nodeJS has one thread)

Re: Show HN: Pyper – Concurrent Python Made Simple

#26

Post by zenapollo »

JackC wrote: post #21

Cool, I was just looking for something like this! It's surprisingly annoying in built-in python to do something like this. The most recent thing I was trying to do was: - load URLs from a file - hand them out to one subprocess per cpu - download them concurrently in threads or async within each subprocess - pull the results back into a single process for formatting and storing Getting this to work and handle queues,…

I stumbled on grequests for this use case and it just works.

Re: Show HN: Pyper – Concurrent Python Made Simple

#27

Post by dec0dedab0de »

giancarlostoro wrote: post #14

Lowkey I hate the "\" line continuation in Python to force PEP-8 compliance in a way... Is there any Pythonistas who would write the examples in there differently to achieve a similar level of readability? > pipeline = task(get_data, branch=True) \ > | task(step1, workers=20) \ > | task(step2, workers=20) \ > | task(step3, workers=20, multiprocess=True)

I really don’t like overloading pipes like this. I would rather chain methods like how the django orm does it.

you could reassign every line, but it would look nicer with chained functions.

  pipeline = task(get_data, branch=True)
  pipeline = pipeline | task(step1, workers=20)
  pipeline = pipeline |  task(step2, workers=20)
  pipeline = pipeline |  task(step3, workers=20, multiprocess=True)
edit:

I would be tempted to do something like this:

  steps = [task(step1, workers=20),
           task(step2, workers=20),
           task(step3, workers=20, multiprocess=True)]
  pipeline = task(get_data, branch=True)

  for step in steps:
      pipeline =   pipeline.__or__(step)

Re: Show HN: Pyper – Concurrent Python Made Simple

#29

Post by Rickster35 »

dec0dedab0de wrote: post #27
giancarlostoro wrote: post #14

Lowkey I hate the "\" line continuation in Python to force PEP-8 compliance in a way... Is there any Pythonistas who would write the examples in there differently to achieve a similar level of readability? > pipeline = task(get_data, branch=True) \ > | task(step1, workers=20) \ > | task(step2, workers=20) \ > | task(step3, workers=20, multiprocess=True)

I really don’t like overloading pipes like this. I would rather chain methods like how the django orm does it. you could reassign every line, but it would look nicer with chained functions. pipeline = task(get_data, branch=True) pipeline = pipeline | task(step1, workers=20) pipeline = pipeline | task(step2, workers=20) pipeline = pipeline | task(step3, workers=20, multiprocess=True) edit: I would be tempted to do som…

According to the docs, | is syntactic sugar for the .pipe method.

  pipeline = task(get_data, branch=True).pipe(
      task(step1, workers=20)).pipe(
      task(step2, workers=20)).pipe(
      task(step3, workers=20, multiprocess=True))
That's probably the chained method approach for those with this preference.

Re: Show HN: Pyper – Concurrent Python Made Simple

#30

Post by solidasparagus »

cess11 wrote: post #19
globular-toast wrote: post #10

Earlier quoted context omitted.

Do you really need to reinvent the wheel every time for parallel workloads? Just learn GNU parallel and write single-threaded code. Concurrency in general isn't about parallelism. It's just about doing multiple things at the same time.

GNU Parallel is really neat, software that's so good it's boring. Closing in on being a quarter century old by now, no? I remember first reading about it in 2003 maybe? I've also used 'fork in Picolisp a lot for this kind of thing, and also Elixir, which arguably has much nicer pipes. But hey, it's good that Python after like thirty years or so is trying to get decent concurrency. Eventually people that use it as a f…

Why did your comment need to be so condescending?
Post reply on HN