Live data from Hacker News

Overhead of Python asyncio tasks

textual.textualize.io

11–20 of 88 posts

Re: Overhead of Python asyncio tasks

#11

async tasks are cool, but the usual PSA applies here: Be careful to hold your references, because async tasks without active references will be garbage collected. I've been bitten by that in the past. Long discussion here: https://bugs.python.org/issue21163 Docs: https://docs.python.org/3/library/asyncio-task.html#asyncio.... "Important Save a reference to the result of this function, to avoid a task disappearing mid…

Thanks for this!

Re: Overhead of Python asyncio tasks

#12

After running that code on both a Windows SB3 and major souped up Lenovo running Ubuntu...I just feel inadequate.

M2 Macbook Pro 16GB 16-inch 2023

100,000 tasks 184,167 tasks per/s

200,000 tasks 160,964 tasks per/s

300,000 tasks 165,278 tasks per/s

400,000 tasks 149,577 tasks per/s

500,000 tasks 160,593 tasks per/s

600,000 tasks 168,098 tasks per/s

700,000 tasks 161,837 tasks per/s

800,000 tasks 160,364 tasks per/s

900,000 tasks 149,479 tasks per/s

1,000,000 tasks 155,919 tasks per/s

Re: Overhead of Python asyncio tasks

#13

Reading this is like reading early Renaissance alchemist arguing about how much mercury they need to combine with how much silver to create gold... This is so far gone I don't even know where to begin... > It may be IO that gives AsyncIO its name, but Textual doesn't do any IO of its own. So why on Earth are you using AsyncIO? You don't need it, if that's true... > Those tasks are used to power message queues How are…

asyncio is really not all about I/O though, despite the name. You can just as easily use it for concurrency by interleaving tasks with each other. You can imagine multiple UI elements that all need to make progress, and using coroutines to have them cooperatively yield to each other and not just have one element block progress everywhere else.

It's just using asyncio as a task scheduler, nothing more, nothing less. Maybe when you visit Pythonland you might instead be shocked in a more positive way :-)

Re: Overhead of Python asyncio tasks

#14
Why would you want a terminal emulator anywhere near python? Using python for lightweight system utility gui apps seems like using a hammer to screw in a nail. Yeah you can do it and modern hardware is fast enough that you probably won't care, but why??

Re: Overhead of Python asyncio tasks

#15

Reading this is like reading early Renaissance alchemist arguing about how much mercury they need to combine with how much silver to create gold... This is so far gone I don't even know where to begin... > It may be IO that gives AsyncIO its name, but Textual doesn't do any IO of its own. So why on Earth are you using AsyncIO? You don't need it, if that's true... > Those tasks are used to power message queues How are…

This reply strongly reads of 'im smarter than eveyone' and 'know everything.' Is it possible that the author is doing something that you don't know about? Such as writing and reading text to sockets (which would be I/O.) And is it possible they may have a reason to be using asyncio in Python (which uses a single main thread and hence needs something like asyncio for concurrency.)

>Needless to say that the whole benchmark is worthless

Overhead startup isn't worthless.

'I mean, I know, in Pythonland this is just your average Wednesday'

and here we go. The whole post was really just you trying to make out that you're superior to everyone else. In this case looking down on an entire ecosystem. Why? Python is one of the most popular languages. It has an elegant syntax and it's capable of solving most problems. Go jerk yourself off in private.

Re: Overhead of Python asyncio tasks

#16

After running that code on both a Windows SB3 and major souped up Lenovo running Ubuntu...I just feel inadequate.

Above anything, this shows the performance gains from 3.10 -> 3.11:

  >> python3.10 create_task_overhead.py
  100,000 tasks   185,694 tasks per/s
  200,000 tasks   165,581 tasks per/s
  300,000 tasks   170,857 tasks per/s
  400,000 tasks   159,081 tasks per/s
  500,000 tasks   162,640 tasks per/s
  600,000 tasks   158,779 tasks per/s
  700,000 tasks   161,779 tasks per/s
  800,000 tasks   179,965 tasks per/s
  900,000 tasks   160,913 tasks per/s
  1,000,000 tasks  162,767 tasks per/s

  >> python3.11 create_task_overhead.py
  100,000 tasks   289,318 tasks per/s
  200,000 tasks   265,293 tasks per/s
  300,000 tasks   266,011 tasks per/s
  400,000 tasks   259,821 tasks per/s
  500,000 tasks   251,819 tasks per/s
  600,000 tasks   267,441 tasks per/s
  700,000 tasks   251,789 tasks per/s
  800,000 tasks   254,303 tasks per/s
  900,000 tasks   249,894 tasks per/s
  1,000,000 tasks  266,581 tasks per/s

Re: Overhead of Python asyncio tasks

#17

Reading this is like reading early Renaissance alchemist arguing about how much mercury they need to combine with how much silver to create gold... This is so far gone I don't even know where to begin... > It may be IO that gives AsyncIO its name, but Textual doesn't do any IO of its own. So why on Earth are you using AsyncIO? You don't need it, if that's true... > Those tasks are used to power message queues How are…

Python asyncio is actually a co-operative multitasking system, think real-time operating system type stuff. It doesn't do real concurrency but it gives you a nice interface for reasoning about tasks (think thread) while being able to explicitly control when/where they yield control of the event loop, how often they run, etc.

>How are your message queues not doing I/O?

We generally wouldn't consider in-process moving data around to be "I/O", now if you started interacting with an external database/file/pipe/MMAP-ed-file/etc than that would be "I/O".

>What on Earth are they doing then?

Tasks. Kind of like processes but lighter weight. Running an event loop, dispatching signals, that sort of thing. You can do all that by manually writing your own event loop but python's asyncio (IMO) makes it easier to reason about exactly when your yielding the event loop to some other task, and makes it easier to write code that can yield control of the event loop at arbitrary places. So like if you want to update a widget every 10 seconds you can write something like

    while True:
        await asyncio.sleep(10) #Other tasks can run during this 10 seconds sleep
        #Update widget contents
Useful for stuff that needs to periodically poll data, like a process monitor, or even for just simple clock widgets.

---

As an aside that cooperative multitasking can be really nice in micropython, where you can write tight loops in straight assembly if you need to and still get a pleasant task interface for managing higher level tasks/threads. Combined with some interrupt handlers it makes a pretty elegant real-time-ish operating system. (You probably need to manually deal with garbage collections though)

Re: Overhead of Python asyncio tasks

#18

Reading this is like reading early Renaissance alchemist arguing about how much mercury they need to combine with how much silver to create gold... This is so far gone I don't even know where to begin... > It may be IO that gives AsyncIO its name, but Textual doesn't do any IO of its own. So why on Earth are you using AsyncIO? You don't need it, if that's true... > Those tasks are used to power message queues How are…

Message queues can be implemented in memory, so no network or disk I/O. In fact, it seems this project does use the built-in "queue" library: https://docs.python.org/3/library/queue.html

Re: Overhead of Python asyncio tasks

#20
post #14

Why would you want a terminal emulator anywhere near python? Using python for lightweight system utility gui apps seems like using a hammer to screw in a nail. Yeah you can do it and modern hardware is fast enough that you probably won't care, but why??

Python is a really good scripting language? I guess the big alternatives would be C and BASH and I'd pick python for short utilities over both of those any day.

How slow do you think python is?

Post reply on HN