I've also have come full circle on N:M / green threads. Now I'm back to thinking they aren't worth the effort / pain. My experience comes from C / C++ land and not Go and Rust but I think same lessons apply. In experience I ended up using a few different framework from libcoro to Mordor to raw swapcontext(). I was initially sold on the N:M model as a means of having event driven programming without the callback hell.…
I'm working on my own userspace concurrency framework (early draft intro: http://team.s4y.us/ ) and having fun with this stuff. > A lot of 3rd party code doesn't work great with userspace threads. Agreed, especially since third-party code often calls standard library functions (`connect()`, `read()`, etc.) instead of the magic ones that are aware of your concurrency model. At OkCupid, we have our own database driver…
Looking at your project, did you ever take a look at the Mordor C++ library (https://github.com/mozy/mordor)? It looks like it predates your project by a quite a bit of time.
It looks like you have a fun project on your hands and a lot of learning.
My word of caution about having different scheduler for network / disk IO is that you end up creating latency. Depending on what you're doing this may or may not be an issue in your applications. If you're sensitive to that a large number of small size IO request will quickly do this. Then you're stuck trying to optimize this away (trading problems).
If you're edge application does filesystem calls via some network abstraction you might be in a better place. If you're doing a network filesystem I recommend you check out Ceph. It both provides an in kernel Client (shameless plug I contributed code code there) which is nice because you can rely on the OS do cache things for you. It also provides a client library and the underlying RADOS object store library has non-blocking access to it. Check it out.