51–60 of 63 posts
The original nginx is a rare example of what is the best in software engineering - deep understanding of principles and almost Asperger's attention to details (which is obviously good). Its success is justified.
Earlier quoted context omitted.
setjmp()/longjmp() will work, but they're sort of inefficient as, at least under POSIX, they will save and restore the signal mask, which makes for two round-trips to the kernel just for a coroutine context switch. My web server uses coroutines, and for x86 and x86-64 it uses open-coded assembly routines to yield/resume, with fallbacks to setjmp()/longjmp() on other architectures. It works fairly well, performance-wi…
Given the various efforts like libtask, lthread, boost coroutines, etc, it seems like the low-level assembly trickery ought to be isolated and standardized. Maybe some new methods (similar to the setcontext family) should be proposed to the glibc project. Something not at risk of deprecation.
The Boost Coroutine library is built on top, adds type safety, ensures the stack is unwound when contexts are destroyed, and enables propagation of exceptions across switches.
$ ls -lh /usr/lib/libboost_context.so.1.58.0
-rwxr-xr-x 1 root root 55K May 30 09:58 /usr/lib/libboost_context.so.1.58.0
[1] http://www.boost.org/doc/libs/1_58_0/libs/context/doc/html/c...
[2] http://www.boost.org/doc/libs/1_58_0/libs/context/doc/html/c...
Earlier quoted context omitted.
The holy grail of high-availability isn't upgrading stateless software. It's upgrading stateful ones. Like upgrading when a data structure changes between versions. The HTTP protocol nginx serves is stateless and by comparison far simpler. Same goes for Erlang. It offers nothing more than simple function replacement, and that's not enough to handle data structure changes either.
In Erlang, if you need a new data structure for your state, you can check if your state is old, and upgrade it and continue on. In a gen_server, you might have something like handle_call(Request, From, State) when is_record(State, state) -> handle_call(Request, From, upgrade_state(State); handle_call(Request, From, State) when is_record(State, state2) -> ... (you'll want to do something similar on handle_cast and han…
Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency.
http://en.wikipedia.org/wiki/Dynamic_software_updating#Updat...
Erlang doesn't offer this check.
"Old code may still be evaluated because of processes lingering in the old code."
http://www.erlang.org/doc/reference_manual/code_loading.html...
Earlier quoted context omitted.
Read Adam Dunkels' paper on Protothreads.
Thanks for your comment. I don't know how to feel about Protothreads. It's just syntatic sugar for using a state-machine to provide continuations. You're also not allowed to carry state over between continuations (although I can see how you can extend it to carry-over some struct of data or something). This greatly diminishes the usefulness of Protothreads.. so it feels more like a fun proof-of-concept. Are there any…
I've never used Protothreads before, but Contiki (the tiny operating system written by the same person) uses them for its process implementation.
Similar technique is also used by Putty as well: http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html
One thread per CPU, and non-blocking I/O, that's sounds like the usual way to approach the problem. I'm surprised it uses state machines to handle the non-blocking I/O, because modern software engineering provides much more pleasant approaches such as using coroutines.
Coroutines have the distinct disadvantage of needing a stack, much like threads. So-called 'stackless' coroutines aren't really so different to computed gotos in a state machine
It's unrelated to gotos in a state machine.
Having a stack per coroutine is not a big deal, especially if the coroutine library regularly advises the kernel on the memory areas it isn't using (madvise).
What's stored on the stack usually needs to be stored somewhere else and ends up using a similar account of memory.
When you allocate a 1MB stack per coroutine, the kernel will not wire it all to ram, but only the pages that have been touched. When the stack shrinks back and the coroutine yields, the scheduler can call madvise and inform the kernel that the no longer used pages from the 1MB can be reclaimed.
Earlier quoted context omitted.
In Erlang, if you need a new data structure for your state, you can check if your state is old, and upgrade it and continue on. In a gen_server, you might have something like handle_call(Request, From, State) when is_record(State, state) -> handle_call(Request, From, upgrade_state(State); handle_call(Request, From, State) when is_record(State, state2) -> ... (you'll want to do something similar on handle_cast and han…
That's not guaranteed to be safe. Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency. http://en.wikipedia.org/wiki/Dynamic_software_updating#Updat... Erlang doesn't offer this check. "Old code may still be evaluated because of processes lingering in the old code." http://www.erlang.org/doc/reference_ma…
http://www.erlang.org/doc/man/gen_server.html#Module:code_ch...
For example:
http://stackoverflow.com/questions/1840717/achieving-code-sw...
BTW, You can even support downgrade. :)
Earlier quoted context omitted.
setjmp()/longjmp() will work, but they're sort of inefficient as, at least under POSIX, they will save and restore the signal mask, which makes for two round-trips to the kernel just for a coroutine context switch. My web server uses coroutines, and for x86 and x86-64 it uses open-coded assembly routines to yield/resume, with fallbacks to setjmp()/longjmp() on other architectures. It works fairly well, performance-wi…
Given the various efforts like libtask, lthread, boost coroutines, etc, it seems like the low-level assembly trickery ought to be isolated and standardized. Maybe some new methods (similar to the setcontext family) should be proposed to the glibc project. Something not at risk of deprecation.
[1] https://lwn.net/Articles/605607/
(Also, I ended up mixing up ucontext.h with setjmp.h in my comment above; Lwan uses ucontext.h as a fallback. There are coroutine implementations that will use setjmp/longjmp, or at the very least reuse the jmp_buf struct and roll their own asm, though.)
Earlier quoted context omitted.
In Erlang, if you need a new data structure for your state, you can check if your state is old, and upgrade it and continue on. In a gen_server, you might have something like handle_call(Request, From, State) when is_record(State, state) -> handle_call(Request, From, upgrade_state(State); handle_call(Request, From, State) when is_record(State, state2) -> ... (you'll want to do something similar on handle_cast and han…
That's not guaranteed to be safe. Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency. http://en.wikipedia.org/wiki/Dynamic_software_updating#Updat... Erlang doesn't offer this check. "Old code may still be evaluated because of processes lingering in the old code." http://www.erlang.org/doc/reference_ma…
If we're talking about a gen_server, the state is per process, and once the process has switched to the new code, it won't go back, so there's no problem with old code and new state. In non gen_server code, you do need to be careful about when you hit a boundary that gets you into new code; you'd typically want it to be your process's main loop, since that usually tail recurses and doesn't leave a stack in the old code. It is difficult to reason about a situation where you call into new code, and that returns to old code; it's much better to avoid it.
The concurrent case is OK too, each process manages its own state, and upgrades it when it switches to new code. Are you thinking about changes to messages that are being passed and/or global state? In that case, like with any distributed system, you need to load in stages: first load code that can handle old and new messages, then trigger sending new messages (code load or config setting), then load code that only handles new messages.
Earlier quoted context omitted.
That's not guaranteed to be safe. Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency. http://en.wikipedia.org/wiki/Dynamic_software_updating#Updat... Erlang doesn't offer this check. "Old code may still be evaluated because of processes lingering in the old code." http://www.erlang.org/doc/reference_ma…
> Having new code check if your state is old and upgrading isn't enough. You also need to check old code doesn't process new state. That becomes harder under concurrency. If we're talking about a gen_server, the state is per process, and once the process has switched to the new code, it won't go back, so there's no problem with old code and new state. In non gen_server code, you do need to be careful about when you h…
An upgrade doesn't involve only the in-memory state per process though. It also involves state outside the process, like state on disk. Even if each process upgrades it's own state (I'm assuming the gen_server isn't limited to in-memory state; I don't know), an old process accessing from disk a data structure that differs from the one used by the new process isn't safe. You can't just upgrade old processes in stages.
An upgrade can also involve multiple processes. It's hard to upgrade all of them at once. As you mentioned, in the hardest case of all, a distributed system, loading in stages may be the only option, provided the system was explicitly designed such that old and new processes can coexist without safety issues.