Rationale: Or why am I bothering to rewrite nanomsg?

This is at least partially ill-advised. Comes off as an expertly done, but same old refactoring project that could be titled "I didn't understand this and would understand it better if I were designed based on my personal preferences". This is reinforced by the probably approaching 1 that nobody needs another "six of one half dozen of the other" message queue framework, and alluding to an belief that the C++ library is somehow too bloated for embedded environments. While that was at a time true, no reasonably modern embedded system that requires multi-threading to "100s" of threads, or uses 100s of live sockets for message queue I/O, has to avoid C++ for being too heavy. This is just ZeroMQ alternative #N - not anything objectively better, and certainly not "nano" for embedded systems. The "one true messaging framework" is a unicorn - everyone feels like it should exist but nobody can make it.

But for many cases this is not necessary. A simple callback mechanism would be far better, with the FDs available only as an option for code that needs them. This is the approach that we have taken with nng.

Replacing a state machine with callbacks... something something something you're gonna have a bad time. Esp. considering the gripes are about readability, following control flow, and race conditions. Callbacks are objectively worse for all of those things. Control flow is hard to read in state machine frameworks because the primary flow is dictated by something like "nextState(thisState, action)", so you can't follow it with code lookup.

The problem here (and almost always) is lack of documentation or visualization (or picking the wrong abstraction level for the formally defined states). The beauty is that the definition is almost by default naturally easy to parse (tables of states in a header file, etc.). It takes some extra effort, it is a one shot to write something that generates graphviz state chars or something similar from the state table definition. You could write a custom dot syntax generator from a C-style table definition in what, three hours? Doxygen already does this for much more complicated stuff. Googling reveals this is nothing new: https://gist.github.com/freqlabs/24d88ad8e687891c970a69f16f1...

All that said, State Machines are (currently) the one true abstraction for a given program because that is what a computer is, and every program is, to begin with. If you're not using them explicitly, it just means you have a poorly defined/documented state machine. Maybe someday there will be a better model of computing, or a better way to model programs. For now, the human brain isn't getting any better at keeping track of computer programs, and anything more than single-threaded functional-style code is almost certainly not any "absolute" improvement in readability.

I firmly believe that visualization has become necessary due to complexity, and it is past time to embrace it. There is some stigma that visualization is for fakers - "real programmers only use a bare text editor" - or that it is for children learning programming, or non-engineering folks that need pictures because they're dumb. To be a bit hyperbolic, if we want any chance at keeping up with "the machines", we're going to need a better general-purpose, more workable abstraction than text files. There is no more canonical example of this than the issues pointed out here - state machines are the right abstraction for complex systems, but complex state machines are incredibly difficult to follow in source code.

> OpenSSL has it’s own struct BIO for this stuff, and I could not see an easy way to convert nanomsg's usock stuff to accomodate the struct BIO.

It's actually quite easy to write a custom BIO to work with your own state machines and buffering. This could have been a 1-day project....

> There is some stigma that visualization is for fakers - "real programmers only use a bare text editor"

That stigma would have gone away long ago if the actually existing visual languages were less wretched. (I'm looking at you LabView!). No doubt there are better languages than LabView, but while I think a visual language can be good, creating one will involve many as-yet-unknown unknowns. So even good attempts will be toys at first.

A good stepping stone would be visual analysis of textual programs. I want a debugger for a data-flow graph where following edges answers the question "what caused this value to be what it is". And I want to visualise that graph as well as possible, so that I can get an overview of the different possible causes.

Original author of zmq/nanomsg here.

After all those years dealing with the problem of implementing network protocols I believe that this entire tangle of problems exists because we are dealing with something like 35 years of legacy in two different but subtly interconnected areas: concurrency/parallelism and network programming APIs.

The area of concurrency/parallelism started quite reasonably with the idea of processes. But then, at some point, people felt that processes are too heavy-weight and introduced threads (I'm still trying to find out who the culprit is, but it looks like they've covered their tracks well.) When even threads became too heavy-weight people turned to all kinds of callback-driven architectures, state-machine-driven architectures, coroutines, goroutines etc. Now we have all of those gand we are supposed to make them work together flawlessly, which is a doomed enterprise from the beginning.

At the side of network programming, BSD sockets (introduced in 1983) are the only common ground we have. They are long past their expiry date, they don't adapt to many use cases, but there's no alternative. There are more modern APIs there, but, AFAICS, none of them provides enough added value of top to become the new universal standard.

It should be also said that creation of new universal APIs is hindered by a host of weird network protocol designs out there in the wild. The API designer faces a dilemma: either they go for sane API and rule at least some weird protocols out or they try to support everything and end up with one mess of an API. Not a palatable choice to make.

Then there's the area where the two problems about interact. Originally, you were supposed to listen for incoming TCP connections, fork a new process for each one and access the socket using simple single-threaded program with no state machines, using only blocking calls. Today, you are supposed to have a pool of worker threads, listen on file descriptors using poll/epoll/kqueue, then schedule the work to the worker pool by hand. This raises the complexity of any network protocol implementation by couple of orders of magnitude. Also, you get a lot of corner cases, undefined behaviour, especially at shutdown, weird performance characteristics and I am not even speaking of the increased attack surface.

All in all, it's a miracle that with the tools we have we are able to write any network applications at all.

These days I am working on attacking the issue on both fronts. On the concurrency side it's http://libdill.org -- esentially not very interesting, just an reimplementation of goroutines for C, however, what's worth looking at is the idea of "structured concurrency", a system of managing the lifetimes of coroutines in a systemic manner: http://libdill.org/structured-concurrency.html

On the other front, the network programming, I am trying to put together a proposal for revamp of BSD socket API. The goal is to make it possible to layer many protocols on top of each other as well as one alongside the other. It's a work in progress, so take it with a grain of salt: https://raw.githubusercontent.com/sustrik/dsock/master/rfc/s...

Ironically, some disassemblers (most famously IDA) offer a kind of graph view for visual analysis.

Also, IntelliJ - which is fantastic - has had data flow analysis for years: https://blog.jetbrains.com/idea/2009/08/analyzing-dataflow-w...