21–30 of 88 posts
Any good resources to clear things up?
What a fun experiment. I quick converted it to Go using goroutines and waitgroups for fun: https://gist.github.com/schmichael/1a417808b8e88b684838ae9f4...
Edit: Updating per request below.
Tested on a TR2950x. I wonder if NUMA issues in my case or bad python version (3.9.4).
Python
100,000 tasks 77,108 tasks per/s
200,000 tasks 69,945 tasks per/s
300,000 tasks 72,453 tasks per/s
400,000 tasks 74,636 tasks per/s
500,000 tasks 66,253 tasks per/s
600,000 tasks 77,576 tasks per/s
700,000 tasks 69,673 tasks per/s
800,000 tasks 68,176 tasks per/s
900,000 tasks 73,846 tasks per/s
1,000,000 tasks 68,013 tasks per/s
100000 Tasks 523000 Tasks/s
200000 Tasks 550000 Tasks/s
300000 Tasks 550000 Tasks/s
400000 Tasks 559000 Tasks/s
500000 Tasks 547000 Tasks/s
600000 Tasks 539000 Tasks/s
700000 Tasks 547000 Tasks/s
800000 Tasks 540000 Tasks/s
900000 Tasks 560000 Tasks/s
1000000 Tasks 542000 Tasks/sEach of those task_create calls is roughly 10,000,000,000 / 250,000 = 40,000 instructions.
Thats 40'000 instructions of pure overhead as it does not contribute to the task at hand (accidental complexity).
async tasks are cool, but the usual PSA applies here: Be careful to hold your references, because async tasks without active references will be garbage collected. I've been bitten by that in the past. Long discussion here: https://bugs.python.org/issue21163 Docs: https://docs.python.org/3/library/asyncio-task.html#asyncio.... "Important Save a reference to the result of this function, to avoid a task disappearing mid…
This prevents tasks from being garbage collected, but also prevents situations where components can create tasks that outlive their own lifetime. Plus, it's a saner approach when dealing with exception handling and cancellation.
Reading this is like reading early Renaissance alchemist arguing about how much mercury they need to combine with how much silver to create gold... This is so far gone I don't even know where to begin... > It may be IO that gives AsyncIO its name, but Textual doesn't do any IO of its own. So why on Earth are you using AsyncIO? You don't need it, if that's true... > Those tasks are used to power message queues How are…
Textual is a framework for building desktop apps.
I assume the message queues are in-memory structures used to pass messages between tasks, hence no I/O.
This is really fairly standard stuff.
I understand you may not be familiar with GUI software and/or the Python ecosystem but jumping straight to condescension when you don't understand something is not really a good attitude.
async tasks are cool, but the usual PSA applies here: Be careful to hold your references, because async tasks without active references will be garbage collected. I've been bitten by that in the past. Long discussion here: https://bugs.python.org/issue21163 Docs: https://docs.python.org/3/library/asyncio-task.html#asyncio.... "Important Save a reference to the result of this function, to avoid a task disappearing mid…
Async Python is still confusing af - when do I need it, what happens under the hood, does it actually help with performance, sometimes the GIL comes into play and sometimes it doesn't, why do we ever use threads at all if there's a GIL, why is it called async io if we can use it for anything. My mind is kind of scattered and people seems to be using a lot of async Python for some reason. Any good resources to clear t…
It doesn't necessarily improve performance. It's just a much easier way to do non-blocking I/O (note that blocking and threaded I/O is easier to do but much much heavier).
So, 250,000 per second on a (roughly 10,000 MIPS I7 core) Each of those task_create calls is roughly 10,000,000,000 / 250,000 = 40,000 instructions. Thats 40'000 instructions of pure overhead as it does not contribute to the task at hand (accidental complexity).
edit: I'm not actually sure you are counting the context switch, but I still don't think estimating instruction count that way is particularly useful.
What a fun experiment. I quick converted it to Go using goroutines and waitgroups for fun: https://gist.github.com/schmichael/1a417808b8e88b684838ae9f4...
I am seeing 500~600k tasks/second in .NET6 w/ TPL. Edit: Updating per request below. Tested on a TR2950x. I wonder if NUMA issues in my case or bad python version (3.9.4). Python 100,000 tasks 77,108 tasks per/s 200,000 tasks 69,945 tasks per/s 300,000 tasks 72,453 tasks per/s 400,000 tasks 74,636 tasks per/s 500,000 tasks 66,253 tasks per/s 600,000 tasks 77,576 tasks per/s 700,000 tasks 69,673 tasks per/s 800,000 ta…
EDIT. My results on a 5950x (undervolted)
python3.8.exe test.py
100,000 tasks 139,130 tasks per/s
200,000 tasks 121,905 tasks per/s
300,000 tasks 120,000 tasks per/s
400,000 tasks 114,286 tasks per/s
500,000 tasks 119,403 tasks per/s
600,000 tasks 117,073 tasks per/s
700,000 tasks 130,612 tasks per/s
800,000 tasks 122,488 tasks per/s
900,000 tasks 120,000 tasks per/s
1,000,000 tasks 110,155 tasks per/s
python3.11.exe .\test.py
100,000 tasks 206,452 tasks per/s
200,000 tasks 185,507 tasks per/s
300,000 tasks 186,408 tasks per/s
400,000 tasks 179,021 tasks per/s
500,000 tasks 167,539 tasks per/s
600,000 tasks 177,778 tasks per/s
700,000 tasks 188,235 tasks per/s
800,000 tasks 180,919 tasks per/s
900,000 tasks 168,421 tasks per/s
.\test.exe (go 1.20 compiled)
100000 tasks 2710563.336378 tasks per/s
200000 tasks 3076885.207567 tasks per/s
300000 tasks 3332292.917434 tasks per/s
400000 tasks 3040479.422795 tasks per/s
500000 tasks 2810232.844653 tasks per/s
600000 tasks 3004138.200371 tasks per/s
700000 tasks 2738877.029117 tasks per/s
800000 tasks 2893730.985022 tasks per/s
900000 tasks 3043877.494077 tasks per/s
1000000 tasks 2857992.089078 tasks per/s