Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Offset: Goroutines ported to Python

docs.google.com

Post reply on HN
Print view View original

1–10 of 14 posts

Re: Offset: Goroutines ported to Python

#3

Post by comex »

Mildly related - it's been my opinion that Go's M:N threading system should be considered a workaround for a missing OS feature: the ability to manage and schedule native threads in userspace, without incurring a context switch. There are many reasons why goroutines would be better implemented in cooperation with the kernel - better debugger support, thread local storage, probably less scheduler overhead - but an important one is that it would make them easier to port to other systems than the Go runtime, such as this one. This is not just because they wouldn't have to reimplement the scheduler (which could be addressed with a regular library), but because they wouldn't have to contort every interface that touches syscalls into taking the userland scheduler into account. No need to put syscalls in their own threads or pass the context object around or anything, the kernel would just return to another goroutine while working on a syscall, just like any other thread.

I can dream.

Re: Offset: Goroutines ported to Python

#4

Post by MetaCosm »

comex wrote: post #3

Mildly related - it's been my opinion that Go's M:N threading system should be considered a workaround for a missing OS feature: the ability to manage and schedule native threads in userspace, without incurring a context switch. There are many reasons why goroutines would be better implemented in cooperation with the kernel - better debugger support, thread local storage, probably less scheduler overhead - but an imp…

Not really a work-around, even with great native thread support and tooling as you dream of (and don't get me wrong, I am all for it), ultra-light "goroutines" or "erlang processes" fit a different niche, you are never going to spin up 3+ million OS threads, you absolutely can (and do) in the ultra-light process space.

Re: Offset: Goroutines ported to Python

#6

Post by comex »

MetaCosm wrote: post #4
comex wrote: post #3

Mildly related - it's been my opinion that Go's M:N threading system should be considered a workaround for a missing OS feature: the ability to manage and schedule native threads in userspace, without incurring a context switch. There are many reasons why goroutines would be better implemented in cooperation with the kernel - better debugger support, thread local storage, probably less scheduler overhead - but an imp…

Not really a work-around, even with great native thread support and tooling as you dream of (and don't get me wrong, I am all for it), ultra-light "goroutines" or "erlang processes" fit a different niche, you are never going to spin up 3+ million OS threads, you absolutely can (and do) in the ultra-light process space.

Having 3 million threads doesn't change the usefulness of being able to call into native syscalls and native libraries that use TLS, locks, etc. It does change the usefulness of native debugger support, since existing debuggers would have a rough time with 3 million threads, but it's not like goroutines currently have any solution to that - Go just uses a GDB script to implement goroutine commands similar to the thread ones. So I think it should absolutely be possible to spin up 3 million OS threads. The kernel needs to change to trust memory mapped into userland to keep track of threads rather than tracking all the data structures itself, and the toolchain needs to support split stacks. This would require a lot of change, but the result would be a lot cleaner than having two strongly overlapping concepts of threads.

Re: Offset: Goroutines ported to Python

#9

Post by benoitc »

jzelinskie wrote: post #7

Is this specifically for CPython? I thought Stackless Python already provided something similar barring Go-specific things like channels.

It is also working for Pypy. this is a different impementation from Python Stackless and also give you the possibility to select operations just like in Go.
Post reply on HN