31–40 of 72 posts
We've been happily using Go for a few internal systems are are slowly galvanising most of our backend with it too. Life is happier.
We write a lot of EventMachine code here; I have some questions. He writes: EventMachine is • A frankenstein guts the ruby internals • Not in active development • Makes non-blocking IO block • Requires special code from Ruby libraries • Hard to use in an OOP way • Is really difficult to work with • Poorly documented I'm not sure I understand how EventMachine "guts the Ruby internals" (I didn't watch the talk). It's tr…
> I'm not sure I understand how EventMachine "guts the Ruby internals" EventMachine does not use the Ruby IO primitives (e.g. TCPSocket, UDPSocket, etc) as the basis of its IO abstraction, and instead has reimplemented its own set of primitives for doing IO. Because of this it can't take advantage of work being done in Ruby core to advance Ruby's socket layer. For this reason IPv6 support langered, among other proble…
* select() call used to segfault. * the non blocking read/write behaved very differently under different OSes and I am not talking Windows. There were differences between Linux and OSX for example. * Again frequent crashes when reading/writing when nonblock flag is set.
I am sure, situation is lot better now but for building a reactor library, native Ruby IO primitives fell short. There is always lack of advanced selectors (Epoll/KQueue) as well.
Now as I replied to thomas(?) below and since you yourself have wrapped libevent for Ruby 1.9, there were severe limitation in interpreter back then for even libevent wrapper to work. So I guess, given historical reasons, it made sense Eventmachine did not use native Ruby IO primitives or libevent.
I've looked at Celluloid. It offers a much nicer paradigm than eventmachine. It also provides a nice way to distribute actors over multiple machines (DCell). I strongly agree with this slidedeck. Callback muck is maddening.
I wonder why nobody ever talks about drb, a distributed system that ships with ruby. It's active in Japan, and there has been an effort to make it better known in the rest of the world with a book dedicated to the topic in english (published by Pragmatic Programmers). Still, it hasn't stirred much interest. Even if it's less advanced than DCell, it's still a recommended way to start learning about distributed computi…
1. https://github.com/celluloid/dcell/commit/e3115f284084a78756...
I have yet to test it in production, but it looks pretty promising.
I had a lot of problems with EM blocking networks connections if an event loop was tying up CPU, but I suppose that's to be expected.
We write a lot of EventMachine code here; I have some questions. He writes: EventMachine is • A frankenstein guts the ruby internals • Not in active development • Makes non-blocking IO block • Requires special code from Ruby libraries • Hard to use in an OOP way • Is really difficult to work with • Poorly documented I'm not sure I understand how EventMachine "guts the Ruby internals" (I didn't watch the talk). It's tr…
Do like the http library does and separate out socket handling to be in terms of net.Listeners/net.Conns, and create a handler abstraction for yourself. Your socket code will be testable, because its trivial to write fake net.Conns that are backed with byte buffers. Your handler code will be testable because they are in terms of parsed objects.
I tend to write socket code once per server project, and then leave it alone for months, so this isn't a big deal for me. What makes Go nice for me is that I can block in client code, and retain the efficiency of using a select loop.
I don't have access to the source for the proxy I wrote for work, but here's one I whipped up quickly: http://play.golang.org/p/Fz19qSehCg
Go doesn't expose select, and has the runtime do that for you; but this allows them to make all Go libraries share a select loop, which has nice performance characteristics. Although, now that I think about it, it is probably possible to have a userspace implementation of select that works atop the runtime's shared select loop. Hmmm...
Earlier quoted context omitted.
Thanks. I think I understand a bit better now. Go provides concurrency primitives (scheduler yielding i/o, channels, goroutines, etc) upon which you can build your own "events". I agree (and this seems to be your point) that Go doesn't provide a canned event mechanism that you can just hook into for callbacks on IO. For example in the Go http server[1] a goroutine accepts in a loop and kicks off goroutines to process…
Sure, I'm using the Go http server. Take the example of a simple proxy to see where I'm coming from. Sure, I'm happy to have Go manage all the connections and sockets and I'm happy to spawn new goroutines for each connection and all that. But a proxy accepts an inbound connection, makes an outbound connection, and then monitors the outbound and inbound sides for data. The loop to do this with sockets could be a simpl…
What does Golang's select/case not do that you want?
Earlier quoted context omitted.
> I'm not sure I understand how EventMachine "guts the Ruby internals" EventMachine does not use the Ruby IO primitives (e.g. TCPSocket, UDPSocket, etc) as the basis of its IO abstraction, and instead has reimplemented its own set of primitives for doing IO. Because of this it can't take advantage of work being done in Ruby core to advance Ruby's socket layer. For this reason IPv6 support langered, among other proble…
I like libevent and all, but I don't think Twisted's use of libevent is a particularly big win for Twisted. Again, this is apocryphal, but I remember someone trying to wrap Ruby around libevent and failing; I remember there being a reason this had to be done bespoke. And having fallen into this particular NIH trap many times before: there's just not a whole lot to a simple socket I/O loop. What's been your experience…
Earlier quoted context omitted.
Sure, I'm using the Go http server. Take the example of a simple proxy to see where I'm coming from. Sure, I'm happy to have Go manage all the connections and sockets and I'm happy to spawn new goroutines for each connection and all that. But a proxy accepts an inbound connection, makes an outbound connection, and then monitors the outbound and inbound sides for data. The loop to do this with sockets could be a simpl…
Is there a large overhead for having to use a goroutine rather than the lower level? Do you think it's appropriate for Go to expose that, or do you just miss it?
No.
Earlier quoted context omitted.
> Golang is militantly anti-event; it doesn't even offer a select primitive s/anti-event/anti-callback/. They are not the same thing. Under the hood, Golang is doing the eventing and select() for you, while letting you write simple procedural code. > I try to force myself to write socket code like I did when I was 13, reading a line, parsing it, and writing its response, but that code is brittle and harder to follow…
Real evented code isn't a series of onRead/onWrite callbacks, because real evented code buffers strategically, and abstracts itself into state machines with simple callback interfaces, so that the rest of the program can be written in terms of higher level events. I don't want to turn this into "events work for every program", because like I said above: I don't think event interfaces work for all kinds of programs. I…
Using non-blocking APIs requires you to turn your application "inside out", putting state that would normally have been on the stack into a state machine. You wind up in a maze of callbacks and low-level details.
This kind of thing might be appropriate for writing high performance code in C, but it's a mystery to me why anyone would want to do it in a higher-level language.
We write a lot of EventMachine code here; I have some questions. He writes: EventMachine is • A frankenstein guts the ruby internals • Not in active development • Makes non-blocking IO block • Requires special code from Ruby libraries • Hard to use in an OOP way • Is really difficult to work with • Poorly documented I'm not sure I understand how EventMachine "guts the Ruby internals" (I didn't watch the talk). It's tr…
What EventMachine does (or at least what it did a year or so ago when I was debugging this) is this: sub processes are equated to popen. When the input side of that pipe closes (that is, when the sub process closes STDOUT) the process will be finally be waited upon—and if it doesn't terminate in a hard coded timeout which by the way blocks the rest of your program, then it will be forcibly gunned down, with SIGKILL if necessary.
Among the problems that arise here, note that unless your daemon script is written to unconditionally drop STDOUT upon forking (uncommon) and you attempt to launch a daemon from within a sub process you are managing using EventMachine, the subprocess itself will terminate quickly, the daemon will go on its merry way, and your driver program will never, ever tell you it has finished running until that daemon is dead and anything it has spawned that might possibly use STDOUT is also dead. And god forbid it close that stream and then dare to continue running, for EventMachine will shoot it dead within IIRC 20 seconds, and lock up your driver program for the duration to boot.
Programmers who understand how the Unix process model works will write a very small signal handler for SIGCHLD that writes a byte on a pipe or some similar method of notifying the main event loop and call wait on the child immediately and then close its end of those pipes. I am reliably informed by those who understand the Windows process model that what EventMachine does is even more wrong there. This is a subsystem that was not written by anyone who knew what popen does, could not be bothered (or was perhaps incompetent to read) what any of a dozen standard implementations of it do, and appears to have debugged the code into some form of submission and then released it upon an unsuspecting public.
This is the only colossally wrong decision they made that I can list off the top of my head, but that's because it was so stupid I stopped looking for trouble after that. EventMachine does not handle anything but a very straightforward select loop very well, and I am sufficiently terrified of what lives under the covers in that system that I would rather write the select by hand (massive pain though it may be) than let this system anywhere near it. The thing that really alarms me is that people build walls of cardboard like NeverBlock (which reaches deeply into the guts of the Ruby software I/O and replaces it with EventMachine driven coroutines) atop this foundation of sand and then wonder when it falls over sideways in an impenetrable and impossible to debug fashion.
Coroutine programming (for that is, essentially, what we are talking about) can be a very elegant way to solve certain problems, but it works best when it is simple, or it least localized (e.g. samefringe). In an event driven server, every little piece must be audited carefully to ensure that it does not block. You get all the same problems any preemptive concurrency model does, with some added nastiness; in exchange you get some slightly better scalability numbers. It is at its best in a fairly simple program such as, say, nginx in its proxy configuration, where it speaks streams and SSL and talks to some application server on the other end of a different stream for anything sophisticated.