Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rationale: Or why am I bothering to rewrite nanomsg?

nanomsg.github.io

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#41

Post by sevensor »

tel wrote: post #40
blattimwind wrote: post #7

I have to admit, after all these years, that I take everything coming from that general direction with a huge grain of salt. Crossroads I/O was supposed to be the great zmq successor and failed entirely, nanomsg was supposed to be an even better redesign of zmq and failed and now nanomsg-ng is supposed to be an even better design iteration on nanomsg. Meanwhile the old/bad/bloated/poorly designed zmq just kept workin…

Because ZMQ is vastly better managed from a human POV. The energy required to make a project like this succeed surpasses the individual.

I think that's the important insight behind the success of ZMQ. Pieter Hintjens understood that a software library had a social as well as a technical context. ZMQ was built to be a delight to work with, not just a technical achievement. I have lots of respect for Sústrik, but I get the impression that he's willing to sacrifice everything else to have the right implementation.

Really, I think it takes both a Hintjens and a Sústrik to make something great. (Or a Lennon and a McCartney, a Jobs and a Wosniack, a Rodgers and a Hart.) There's a tension between giving people something with integrity, and giving people something they'll love.

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#42

Post by otabdeveloper1 »

farazbabar wrote: post #19
dmitrygr wrote: post #9

> even so-called "modern" operating systems > like macOS 10.12 and Windows 8.1 simply > melted or failed entirely when creating > any non-trivial number of threads. (To > me, creating 100 threads should be a no-brainer This is nonsense. NT kernel happily creates thousands of threads. Just made a little test app to try. No issues at all.

Modern operating systems are very efficient at scheduling threads with negligible workloads compared to threads that are performing under load. I suspect that a lot of test code for toy examples and default workload for thousands of threads you see on an operating system has running at any given time is probably not cpu intensive enough to matter. Compared to this, a library designed to perform actual work using hund…

There's no such thing as a "context switching overhead". You don't understand how operating systems work.

The OS context switches to the next available thread at fixed time intervals. It doesn't matter if you have 5 threads or 5000, the number of context switches is the same.

The only difficulty is deciding which thread is the "next available" one. This is the job of the process scheduler.

Normally, the process scheduler is a complex heuristic affair that tries to pick a more deserving thread based on some black magic rules of thumb. Under certain loads this black magic can backfire, picking some threads more frequently while letting others starve.

If you know your workload, you can use a scheduler without magic heuristics. These kinds of schedulers are called "real time". Try them, you'll be surprised.

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#43

Post by viraptor »

tuukkah wrote: post #34
theincredulousk wrote: post #21

This is at least partially ill-advised. Comes off as an expertly done, but same old refactoring project that could be titled "I didn't understand this and would understand it better if I were designed based on my personal preferences". This is reinforced by the probably approaching 1 that nobody needs another "six of one half dozen of the other" message queue framework, and alluding to an belief that the C++ library…

> All that said, State Machines are (currently) the one true abstraction for a given program because that is what a computer is, and every program is, to begin with. I don't understand these claims. To me, a computer "is" processors that step through memory locations interpreting them as operations and operands - not a state machine. Equally, hardly any program is a state machine. Is a Haskell program a state machine…

> To me, a computer "is" processors that step through memory locations interpreting them as operations and operands - not a state machine.

And what is that, if not a state machine with memory? Where the state transitions come from decoding/executing instructions and arguments, and the memory comes from ram/registers.

Although yes, that's kind of getting to a similar theoretical level as object = function.

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#44

Post by JustSomeNobody »

tuukkah wrote: post #34
theincredulousk wrote: post #21

This is at least partially ill-advised. Comes off as an expertly done, but same old refactoring project that could be titled "I didn't understand this and would understand it better if I were designed based on my personal preferences". This is reinforced by the probably approaching 1 that nobody needs another "six of one half dozen of the other" message queue framework, and alluding to an belief that the C++ library…

> All that said, State Machines are (currently) the one true abstraction for a given program because that is what a computer is, and every program is, to begin with. I don't understand these claims. To me, a computer "is" processors that step through memory locations interpreting them as operations and operands - not a state machine. Equally, hardly any program is a state machine. Is a Haskell program a state machine…

> To me

>, a computer "is" processors that step through memory locations interpreting them as operations and operands - not a state machine.

Each of those operations changes the processor's state based on the current state and the input. Which is what a state machine does/is.

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#45

Post by hood_syntax »

sevensor wrote: post #41
tel wrote: post #40

Earlier quoted context omitted.

Because ZMQ is vastly better managed from a human POV. The energy required to make a project like this succeed surpasses the individual.

I think that's the important insight behind the success of ZMQ. Pieter Hintjens understood that a software library had a social as well as a technical context. ZMQ was built to be a delight to work with, not just a technical achievement. I have lots of respect for Sústrik, but I get the impression that he's willing to sacrifice everything else to have the right implementation. Really, I think it takes both a Hintjens…

After reading a lot of writing by both Hintjens and Sústrik I came to the same conclusion. It's such a shame they fell out, I feel like with a little compromise on both sides they could have come to a workable place. There would have been heads butting and voices being raised (figuratively) but a lot of great work can be done in a place like that. The problem is that it's tiring to fight those battles on a daily basis, and it gets old really fast.

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#46

Post by viraptor »

otabdeveloper1 wrote: post #42
farazbabar wrote: post #19

Earlier quoted context omitted.

Modern operating systems are very efficient at scheduling threads with negligible workloads compared to threads that are performing under load. I suspect that a lot of test code for toy examples and default workload for thousands of threads you see on an operating system has running at any given time is probably not cpu intensive enough to matter. Compared to this, a library designed to perform actual work using hund…

There's no such thing as a "context switching overhead". You don't understand how operating systems work. The OS context switches to the next available thread at fixed time intervals. It doesn't matter if you have 5 threads or 5000, the number of context switches is the same. The only difficulty is deciding which thread is the "next available" one. This is the job of the process scheduler. Normally, the process sched…

> There's no such thing as a "context switching overhead".

So what do you call the time spent changing the system internal information about the currently executing thread, switching page tables via cr3 and the invalidation it causes, and the time wasted because of cache misses (although a lot of it can be reclaimed with CPU pinning). Because that's what people normally call "context/thread switching overhead"

> The OS context switches to the next available thread at fixed time intervals. It doesn't matter if you have 5 threads or 5000, the number of context switches is the same.

Not just at fixed intervals. If your threads are doing io work, more threads will switch more often because they will issue io requests forcing waits/yields. If they do pure-cpu work, that doesn't apply, but we're talking about networking mostly in these comments.

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#47

Post by bpicolo »

lmm wrote: post #37
rumcajz wrote: post #32

Earlier quoted context omitted.

One hard requirement here is to be able to expose C API. Given the current state of affairs, it's the only way to guarantee that any language runtime will be able to use it. That limits the options quite a lot. I hear it's possible to make shared libraries with Golang nowadays, so that may be an option. Probably Rust, but I'm not sure what's the state of green-threading there. Finally, if you want to go implementing…

Rust doesn't yet have built-in green-threading support (currently you have to rely on a non-core library), but that's the case in C too and will always be so. So I don't understand why you'd regard C as a better choice than Rust?

They intentionally removed green threading from Rust long ago, so I would guess they intend to leave it outside core?

https://github.com/rust-lang/rfcs/blob/0806be4f282144cfcd55b...

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#48

Post by nfoz »

Do people generally consider nanomsg to be a "failed experiment"? Is anyone using it for their projects?

The author's tone makes it seem like I shouldn't use nanomsg. nanomsg in turn makes it seem like I shouldn't use ZeroMQ.

So what would people recommend me to use right now for a project? Are these issues all that serious?

My intended use would be for a simple friendly pub-sub API for programs to talk to each other, locally or across a network.

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#49

Post by ezdiy »

rumcajz wrote: post #38
Matthias247 wrote: post #26

Earlier quoted context omitted.

> Replacing a state machine with callbacks... something something something you're gonna have a bad time Having written quite a few networking systems, I fully agree. Especially if it's coupled in this text with "replacing a single-threaded state-machine" with "a multithreaded system, which uses callbacks". IMHO the latter is almost guaranteed a recipe for all kinds of multithreading issues, from race-conditions over…

That's interesting. Why do you need state machine if you have goroutine per connection? Isn't the instruction pointer + local variables an equivalent of a state stored in a state machine?

HTTP2 is all about "tcp-inside-tcp" states.

https://github.com/golang/net/blob/master/http2/http2.go#L81

Wherever possible, they do the sensible thing - just goroutines for each flow, piped into from the muxed frame parser via channels, but the state for each flow must be still tracked as per-flow state in the topmost flow dispatcher - goroutine can't tear itself down on timeouts, inspect OOB signalilng of the topmost stream and such.

Re: Rationale: Or why am I bothering to rewrite nanomsg?

#50

Post by lmm »

bpicolo wrote: post #47
lmm wrote: post #37

Earlier quoted context omitted.

Rust doesn't yet have built-in green-threading support (currently you have to rely on a non-core library), but that's the case in C too and will always be so. So I don't understand why you'd regard C as a better choice than Rust?

They intentionally removed green threading from Rust long ago, so I would guess they intend to leave it outside core? https://github.com/rust-lang/rfcs/blob/0806be4f282144cfcd55b...

I think the intention is that userspace schedulers (event loops) will remain outside of core, but core will potentially contain a task interface and know how to form tasks (but not necessarily offer any way to execute them asynchronously) so that there can be better integration of ownership/borrow checking with async tasks? But I haven't been following that closely.
Post reply on HN