An easy way to concurrency and parallelism with Python stdlib

Post by paddy_m » Thu, Sep 14, 2023, 5:15 PM UTC

nerdponx wrote: post #31

xeromal wrote: post #19

Earlier quoted context omitted.

Coming from C#, I honestly HATE python's multiprocessing and multithreading. Hell, I hate it's async await. I learned recently that in one mode, it pipes the values across the process and this made it impossible to use when passing along large pandas dataframes. I'm sure half of it is just my own lack of knowledge with python's abilities but C# sure made it easier. lol

For Pandas, I recommend the third-party Joblib library: https://joblib.readthedocs.io/en/latest/

Post by networked » Thu, Sep 14, 2023, 5:50 PM UTC

FreakLegion wrote: post #5

pdimitar wrote: post #4

Earlier quoted context omitted.

Certainly. My point is that if you need to write that much code and/or do that much research, at one point the effort of doing it in another language will be less than to keep insisting on using a tool that's not designed for it. It happened with me and many other former colleagues. Though obviously, everyone decides for themselves when does that point come -- or if it comes at all.

The point of the article is a handful of lines. The rest is accoutrement like the URL list and timing code. But sure, if tasks = {} for url in URLs: future = executor.submit(fetch_url, url) tasks[future] = url bothers you, this is perfectly (some would say more so even than the original) Pythonic: tasks = {executor.submit(fetch_url, url): url for url in URLs}

  def main():
      with ThreadPoolExecutor(max_workers=len(URLs)) as executor:
          for url, title in zip(URLs, executor.map(fetch_url, URLs)):
              print(f"URL: {url}\nTitle: {title}")

  min(32, os.cpu_count() + 4)

  max_workers=len(items_to_process)

Post by hot_gril » Thu, Sep 14, 2023, 6:35 PM UTC

cle wrote: post #17

I recently have been doing--what should be--straightforward subprocess work in Python, and the experience is infuriatingly bad. There are so many options for launching subprocesses and communicating with them, and each one has different caveats and undocumented limitations, especially around edge cases like processes crashing, timing out, killing them, if they are stuck in native code outside of the VM, etc. For exam…

Post by seabrookmx » Thu, Sep 14, 2023, 6:58 PM UTC

hot_gril wrote: post #36

cle wrote: post #17

I recently have been doing--what should be--straightforward subprocess work in Python, and the experience is infuriatingly bad. There are so many options for launching subprocesses and communicating with them, and each one has different caveats and undocumented limitations, especially around edge cases like processes crashing, timing out, killing them, if they are stuck in native code outside of the VM, etc. For exam…

Python manages to combine the worst parts of high-level and low-level programming when it comes to multithreading. Like it's using multiple OS-level threads with the associated overhead (not greenthreading like in JS), except it's locking to negate actual multiprocessing, but you still have to use mutexes about as much as in C (no event loop like JS), and the whole API feels low-level and convoluted. It's like they t…

Post by crabbone » Thu, Sep 14, 2023, 8:18 PM UTC