Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rationale: Or why am I bothering to rewrite nanomsg?

nanomsg.github.io

Post reply on HN
Print view View original

31–40 of 60 posts

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#31

Post by willtim »

rumcajz wrote: post #27

Original author of zmq/nanomsg here. After all those years dealing with the problem of implementing network protocols I believe that this entire tangle of problems exists because we are dealing with something like 35 years of legacy in two different but subtly interconnected areas: concurrency/parallelism and network programming APIs. The area of concurrency/parallelism started quite reasonably with the idea of proce…

> it's a miracle that with the tools we have we are able to write any network applications at all

Perhaps it is time to start using a higher-level language? One with a runtime that provides green threads?

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#32

Post by rumcajz »

willtim wrote: post #31
rumcajz wrote: post #27

Original author of zmq/nanomsg here. After all those years dealing with the problem of implementing network protocols I believe that this entire tangle of problems exists because we are dealing with something like 35 years of legacy in two different but subtly interconnected areas: concurrency/parallelism and network programming APIs. The area of concurrency/parallelism started quite reasonably with the idea of proce…

> it's a miracle that with the tools we have we are able to write any network applications at all Perhaps it is time to start using a higher-level language? One with a runtime that provides green threads?

One hard requirement here is to be able to expose C API. Given the current state of affairs, it's the only way to guarantee that any language runtime will be able to use it.

That limits the options quite a lot. I hear it's possible to make shared libraries with Golang nowadays, so that may be an option. Probably Rust, but I'm not sure what's the state of green-threading there.

Finally, if you want to go implementing protocols further down the stack (L4, L3) the performance requirements get tighter. So, for example, Golang's GC may rule that language out.

Taking all those requirements into account I guess C is still the only option. Maybe Rust one day.

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#33

Post by twic »

rumcajz wrote: post #27

Original author of zmq/nanomsg here. After all those years dealing with the problem of implementing network protocols I believe that this entire tangle of problems exists because we are dealing with something like 35 years of legacy in two different but subtly interconnected areas: concurrency/parallelism and network programming APIs. The area of concurrency/parallelism started quite reasonably with the idea of proce…

1985 called, they want their composable networking abstraction back:

https://en.wikipedia.org/wiki/STREAMS

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#34

Post by tuukkah »

theincredulousk wrote: post #21

This is at least partially ill-advised. Comes off as an expertly done, but same old refactoring project that could be titled "I didn't understand this and would understand it better if I were designed based on my personal preferences". This is reinforced by the probably approaching 1 that nobody needs another "six of one half dozen of the other" message queue framework, and alluding to an belief that the C++ library…

> All that said, State Machines are (currently) the one true abstraction for a given program because that is what a computer is, and every program is, to begin with.

I don't understand these claims. To me, a computer "is" processors that step through memory locations interpreting them as operations and operands - not a state machine. Equally, hardly any program is a state machine.

Is a Haskell program a state machine, or "poorly defined"? I'd suggest other models of computation such as typed lambda calculus provide a much better basis for defining and documenting computer programs.

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#35

Post by rumcajz »

twic wrote: post #33
rumcajz wrote: post #27

Original author of zmq/nanomsg here. After all those years dealing with the problem of implementing network protocols I believe that this entire tangle of problems exists because we are dealing with something like 35 years of legacy in two different but subtly interconnected areas: concurrency/parallelism and network programming APIs. The area of concurrency/parallelism started quite reasonably with the idea of proce…

1985 called, they want their composable networking abstraction back: https://en.wikipedia.org/wiki/STREAMS

Yes, exactly. After all, as Jack Weinberg said, never trust anyone below 30.

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#36

Post by spacenick88 »

rumcajz wrote: post #27

Original author of zmq/nanomsg here. After all those years dealing with the problem of implementing network protocols I believe that this entire tangle of problems exists because we are dealing with something like 35 years of legacy in two different but subtly interconnected areas: concurrency/parallelism and network programming APIs. The area of concurrency/parallelism started quite reasonably with the idea of proce…

So for me the really big question in all of this is "Are threads really too heavyweight?". This obviously needs the constraint "on a sane, modern OS".

For me the most sane C (non-datagram) networking model at least on Linux is threads, each calling accept concurrently (afaik few people know this is supported) and then handling the accepted connection until that is closed. For systems where you only want to handle a fixed number of connections like databases you keep your number of threads fixed for others you start a new thread once every other thread handles a connection already. It gets rid of Thread Pools (since you only do pthread_create() when your number of concurrent connections increases), Async- and callback-hell and makes all your handling code linear.

Every other day I keep seeing "Blabla uses async epoll so handles 10k connections" but a) what serious work can you do with 10k connections i.e. 125 kB/s per connection @ 10 Gbit/s. b) at what cost to readability/maintainability of code and c) are you sure you haven't just moved your bottleneck to something else? Also I've never seen any benchmark showing how this actually beats threads on a modern Linux box.

As for shitty OSs I say fuck them

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#37

Post by lmm »

rumcajz wrote: post #32
willtim wrote: post #31

Earlier quoted context omitted.

> it's a miracle that with the tools we have we are able to write any network applications at all Perhaps it is time to start using a higher-level language? One with a runtime that provides green threads?

One hard requirement here is to be able to expose C API. Given the current state of affairs, it's the only way to guarantee that any language runtime will be able to use it. That limits the options quite a lot. I hear it's possible to make shared libraries with Golang nowadays, so that may be an option. Probably Rust, but I'm not sure what's the state of green-threading there. Finally, if you want to go implementing…

Rust doesn't yet have built-in green-threading support (currently you have to rely on a non-core library), but that's the case in C too and will always be so. So I don't understand why you'd regard C as a better choice than Rust?

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#38

Post by rumcajz »

Matthias247 wrote: post #26
theincredulousk wrote: post #21

This is at least partially ill-advised. Comes off as an expertly done, but same old refactoring project that could be titled "I didn't understand this and would understand it better if I were designed based on my personal preferences". This is reinforced by the probably approaching 1 that nobody needs another "six of one half dozen of the other" message queue framework, and alluding to an belief that the C++ library…

> Replacing a state machine with callbacks... something something something you're gonna have a bad time Having written quite a few networking systems, I fully agree. Especially if it's coupled in this text with "replacing a single-threaded state-machine" with "a multithreaded system, which uses callbacks". IMHO the latter is almost guaranteed a recipe for all kinds of multithreading issues, from race-conditions over…

That's interesting. Why do you need state machine if you have goroutine per connection? Isn't the instruction pointer + local variables an equivalent of a state stored in a state machine?

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#39

Post by AstralStorm »

jesseb wrote: post #16
dmitrygr wrote: post #9

> even so-called "modern" operating systems > like macOS 10.12 and Windows 8.1 simply > melted or failed entirely when creating > any non-trivial number of threads. (To > me, creating 100 threads should be a no-brainer This is nonsense. NT kernel happily creates thousands of threads. Just made a little test app to try. No issues at all.

I raised an eyebrow at that as well. My workstation running Windows 10 is currently chugging along with 236 Processes, 3564 Threads and 154915 Handles, which is pretty typical of this system.

Of those, how many are actually active?

(equivalent to R state in procps)

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#40

Post by tel »

blattimwind wrote: post #7

I have to admit, after all these years, that I take everything coming from that general direction with a huge grain of salt. Crossroads I/O was supposed to be the great zmq successor and failed entirely, nanomsg was supposed to be an even better redesign of zmq and failed and now nanomsg-ng is supposed to be an even better design iteration on nanomsg. Meanwhile the old/bad/bloated/poorly designed zmq just kept workin…

Because ZMQ is vastly better managed from a human POV. The energy required to make a project like this succeed surpasses the individual.
Post reply on HN