Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Aiopandas – Async .apply() and .map() for Pandas, Faster API/LLMs Calls

github.com

Post reply on HN
Print view View original

1–10 of 19 posts

Re: Show HN: Aiopandas – Async .apply() and .map() for Pandas, Faster API/LLMs Calls

#3

Post by eneuman »

gardnr wrote: post #2

This is super useful. So much boilerplate code to run async calls and gather. I have been using tqdm.gather() so I am glad to see this library supports it.

Thanks! I originally built this to scratch an itch I had, so I’m really glad you find it useful too. If you have any ideas for improvements or missing features, feel free to suggest them — or even open a PR!

Re: Show HN: Aiopandas – Async .apply() and .map() for Pandas, Faster API/LLMs Calls

#4

Post by westurner »

Can this be merged into pandas?

Pandas does not currently install tqdm by default.

pandas-dev/pandas//pyproject.toml [project.optional-dependencies] https://github.com/pandas-dev/pandas/blob/8943c97c597677ae98...

Dask solves for various adjacent problems; IDK if pandas, dask, or dask-cudf would be faster with async?

Dask docs > Scheduling > Dask Distributed (local) https://docs.dask.org/en/stable/scheduling.html#dask-distrib... :

> Asynchronous Futures API

Dask docs > Deploy Dask Clusters; local multiprocessing poll, k8s (docker desktop, podman-desktop,), public and private clouds, dask-jobqueue (SLURM,), dask-mpi: https://docs.dask.org/en/stable/deploying.html#deploy-dask-c...

Dask docs > Dask DataFrame: https://docs.dask.org/en/stable/dataframe.html :

> Dask DataFrames are a collection of many pandas DataFrames.

> The API is the same. The execution is the same.

> [concurrent.futures and/or @dask.delayed]

tqdm.dask: https://tqdm.github.io/docs/dask/#tqdmdask .. tests/tests_pandas.py: https://github.com/tqdm/tqdm/blob/master/tests/tests_pandas.... , tests/tests_dask.py: https://github.com/tqdm/tqdm/blob/master/tests/tests_dask.py

tqdm with dask.distributed: https://github.com/tqdm/tqdm/issues/1230#issuecomment-222379... , not yet a PR: https://github.com/tqdm/tqdm/issues/278#issuecomment-5070062...

dask.diagnostics.progress: https://docs.dask.org/en/stable/diagnostics-local.html#progr...

dask.distributed.progress: https://docs.dask.org/en/stable/diagnostics-distributed.html...

dask-labextension runs in JupyterLab and has a parallel plot visualization of the dask task graph and progress through it: https://github.com/dask/dask-labextension

dask-jobqueue docs > Interactive Use > Viewing the Dask Dashboard: https://jobqueue.dask.org/en/latest/clusters-interactive.htm...

https://examples.dask.org/ > "Embarrassingly parallel Workloads" tutorial re: "three different ways of doing this with Dask: dask.delayed, concurrent.Futures, dask.bag": https://examples.dask.org/applications/embarrassingly-parall...

Re: Show HN: Aiopandas – Async .apply() and .map() for Pandas, Faster API/LLMs Calls

#6

Post by eneuman »

westurner wrote: post #4

Can this be merged into pandas? Pandas does not currently install tqdm by default. pandas-dev/pandas//pyproject.toml [project.optional-dependencies] https://github.com/pandas-dev/pandas/blob/8943c97c597677ae98... Dask solves for various adjacent problems; IDK if pandas, dask, or dask-cudf would be faster with async? Dask docs > Scheduling > Dask Distributed (local) https://docs.dask.org/en/stable/scheduling.html#dask…

Thank you for the input! To be honest, I don’t use Dask often, and as a regular Pandas user, I don’t feel the most qualified to comment—but here we go.

Can this be merged into Pandas?

I’d be honored if something I built got incorporated into Pandas! That said, keeping aiopandas as a standalone package has the advantage of working with older Pandas versions, which is useful for workflows where upgrading isn’t feasible. I also can’t speak to the downstream implications of adding this directly into Pandas.

Pandas does not install tqdm by default.

That makes sense, and aiopandas doesn’t require tqdm either. You can pass any class with __init__, update, and close methods as the tqdm argument, and it will work the same. Keeping dependencies minimal helps avoid unnecessary breakage.

What about Dask?

I’m not a regular Dask user, so I can’t comment much on its internals. Dask already supports async coroutines (Dask Async API), but for simple async API calls or LLM requests, aiopandas is meant to be a lightweight extension of Pandas rather than a full-scale parallelization framework. If you’re already using Dask, it probably covers most of what you need, but if you’re just looking to add async support to Pandas without additional complexity, aiopandas might be a more lightweight option.

Re: Show HN: Aiopandas – Async .apply() and .map() for Pandas, Faster API/LLMs Calls

#7

Post by eneuman »

napsternxg wrote: post #5

This is a very clean api and I really like the way you implemented it directly in Pandas. I worked on something similar 2 years back but the API was not as this one. Thanks a lot to making this.

You’re more than welcome! I really appreciate the kind words.

If you have any ideas for improvements, missing features, or run into any issues, don't hesitate to share!

Re: Show HN: Aiopandas – Async .apply() and .map() for Pandas, Faster API/LLMs Calls

#8

Post by refactor_master »

It seems like this hack would be fine for notebooks, but not something I’d be interested in for production code.

Why not just something like this?

  def f(n):
      time.sleep(random.uniform(0.1, 0.3))  # Simulate network delay
      return pd.DataFrame({"A": [n, n+1], "B": [n*2, (n+1)*2]})

  with ThreadPoolExecutor() as ex:
    df = pd.concat(ex.map(f, range(3)), ignore_index=True)

Re: Show HN: Aiopandas – Async .apply() and .map() for Pandas, Faster API/LLMs Calls

#9

Post by isoprophlex »

refactor_master wrote: post #8

It seems like this hack would be fine for notebooks, but not something I’d be interested in for production code. Why not just something like this? def f(n): time.sleep(random.uniform(0.1, 0.3)) # Simulate network delay return pd.DataFrame({"A": [n, n+1], "B": [n*2, (n+1)*2]}) with ThreadPoolExecutor() as ex: df = pd.concat(ex.map(f, range(3)), ignore_index=True)

indeed... the longer i write python, the more i just try to solve stuff with a simple ThreadPoolExecutor.

I think doing this is not the best choice for cpu-bound work, which is likely what you're running into with pandas, but nevertheless... I like how you can almost always slap a threadpool onto something and speed things up, with minimal cognitive overhead.

Re: Show HN: Aiopandas – Async .apply() and .map() for Pandas, Faster API/LLMs Calls

#10

Post by hn8726 »

isoprophlex wrote: post #9
refactor_master wrote: post #8

It seems like this hack would be fine for notebooks, but not something I’d be interested in for production code. Why not just something like this? def f(n): time.sleep(random.uniform(0.1, 0.3)) # Simulate network delay return pd.DataFrame({"A": [n, n+1], "B": [n*2, (n+1)*2]}) with ThreadPoolExecutor() as ex: df = pd.concat(ex.map(f, range(3)), ignore_index=True)

indeed... the longer i write python, the more i just try to solve stuff with a simple ThreadPoolExecutor. I think doing this is not the best choice for cpu-bound work, which is likely what you're running into with pandas, but nevertheless... I like how you can almost always slap a threadpool onto something and speed things up, with minimal cognitive overhead.

> not the best choice for cpu-bound work, which is likely what you're running into with pandas

I'm not a Python user, why is it not good for cpu-bound work? I see the defaults assume some I/O work, but with `max_workers=~cpu_count` it should be what typical dispatchers for CPU-bound work do in other languages

Post reply on HN