31–38 of 38 posts
From just a short skimming of the docs: - my biggest issue with concurrency in python (especially with asyncio) is leaking tasks. Pyper should provide structured concurrency support a-la trio. - I don't see the opposite of branch to collect the output of multiple sub pipelines into a single stage. I need this pretty much always and it is a chore to implement. - Async need not force the full pipeline to be async. Ther…
Nice work! There is a gap when it comes to writing single-machine, concurrent CPU-bound python code. Ray is too big, pykka is threads only, builtins are poorly abstracted. The syntax is also very nice! But I'm not sure I can use this even though I have a specific use-case that feels like it would work well (high-performance pure Python downloading from cloud object storage). The examples are a bit too simple and I do…
One thing I'd mention is that we don't really imagine Pyper as a whole observability and orchestration platform. It's really a package for writing Python functions and executing them concurrently, in a flexible pattern that can be integrated with other tools.
For example, I'm personally a fan of Prefect as an observability platform-- you could define pipelines in Pyper then wrap it in a Prefect flow for orchestration logic.
Exception handling and logging can also be handled by orchestration tools (or in the business logic if appropriate, literally using try... except...)
For a simple progress bar, tqdm is probably the first thing to try. As it wraps anything iterable, applying it to a pipeline might look like:
import time
from pyper import task
from tqdm import tqdm
@task(branch=True)
def func(limit: int):
for i in range(limit):
time.sleep(0.1)
yield i
def main():
for _ in tqdm(func(limit=20), total=20):
pass
if __name__ == "__main__":
main()Nice! I'm looking forward to trying it out. This seems very similar to https://github.com/cgarciae/pypeln/
The important design point we're differing on is that Pyper implements 'pipelines' as functions, whereas pypeln seems to implement 'pipelines' as iterable objects.
From just a short skimming of the docs: - my biggest issue with concurrency in python (especially with asyncio) is leaking tasks. Pyper should provide structured concurrency support a-la trio. - I don't see the opposite of branch to collect the output of multiple sub pipelines into a single stage. I need this pretty much always and it is a chore to implement. - Async need not force the full pipeline to be async. Ther…
Even though there's currently no built-in support for this, a workaround could be to just define synchronous helper functions to handle running your async logic in an event loop.
From just a short skimming of the docs: - my biggest issue with concurrency in python (especially with asyncio) is leaking tasks. Pyper should provide structured concurrency support a-la trio. - I don't see the opposite of branch to collect the output of multiple sub pipelines into a single stage. I need this pretty much always and it is a chore to implement. - Async need not force the full pipeline to be async. Ther…
asyncio.TaskGroup from stdlib provides structured concurrency
Earlier quoted context omitted.
GNU Parallel is really neat, software that's so good it's boring. Closing in on being a quarter century old by now, no? I remember first reading about it in 2003 maybe? I've also used 'fork in Picolisp a lot for this kind of thing, and also Elixir, which arguably has much nicer pipes. But hey, it's good that Python after like thirty years or so is trying to get decent concurrency. Eventually people that use it as a f…
Why did your comment need to be so condescending?
However, it's a real problem that 'beginner languages' like Python and Javascript doesn't readily do multithread computation, something which has been the default on personal computers for quite a while now and available for at least twenty years.
Lowkey I hate the "\" line continuation in Python to force PEP-8 compliance in a way... Is there any Pythonistas who would write the examples in there differently to achieve a similar level of readability? > pipeline = task(get_data, branch=True) \ > | task(step1, workers=20) \ > | task(step2, workers=20) \ > | task(step3, workers=20, multiprocess=True)
I really don’t like overloading pipes like this. I would rather chain methods like how the django orm does it. you could reassign every line, but it would look nicer with chained functions. pipeline = task(get_data, branch=True) pipeline = pipeline | task(step1, workers=20) pipeline = pipeline | task(step2, workers=20) pipeline = pipeline | task(step3, workers=20, multiprocess=True) edit: I would be tempted to do som…
pipeline = task(...)
pipeline |= task(...)
steps = [task(...), task(...)]
pipeline = functools.reduce(operator.or_, steps)
pipeline = pyper.Pipeline([Task(...), Task(...)])