Zed Shaw: "poll, epoll, science, and superpoll" with R • Hacker News Board

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#121

Post by llimllib » Wed, Aug 04, 2010, 4:45 AM UTC

zedshaw wrote: post #66

llimllib wrote: post #42

Earlier quoted context omitted.

> => "I'll pick the best one for the environment" Which, in reality is, "I'll spend a lot of design and implementation effort designing a new one which may or may not improve the measurable, global performance of my new web server because it's not yet at the point where I can benchmark these sorts of things to verify that I'm not wasting a whole ton of effort that could be better spent by deciding that epoll is fast…

It's an idea I had after actually measuring. If it doesn't work then I tried something out. What you really should be getting from it though is that epoll is not faster. It is not O(1). It is not faster on smaller vs. larger lists of FDs. Pretty much all the things you were told as advantages of epoll are total crap. The only advantage of epoll is it's O(N=active) when poll is O(N=total). That's it. So at a minimum I…

I tried to make sure I gave lots of reasons that justify your work; I do think it's cool.

I just wanted to say that it is not an unquestionable design decision.

Rock on with the superpoll, I hope it's awesome and very successful.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#122

Post by kqueue » Wed, Aug 04, 2010, 6:03 AM UTC

FooBarWidget wrote: post #108

kqueue wrote: post #88

Earlier quoted context omitted.

The pointer referred to by the process is not accessible by the kernel because when the user process was running, it had a different vm space than the kernel vm space. So if it just passes the pointer (without copying the pointer's data), then the kernel will point to a virtual address that won't exist until the user process gets swapped in again.

This sounds really strange to me. The kernel has full access to the page tables so can't it lookup things in userspace?

When the kernel is executing a function call placed on the stack, all the addresses on the stack are assumed in the same vm space. It does not know that an address is actually a virtual memory address belonging to process X and tries to figure out the value in the physical memory.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#123

Post by baq » Wed, Aug 04, 2010, 6:15 AM UTC

mfukar wrote: post #37

jacquesm wrote: post #11

Earlier quoted context omitted.

I highly doubt that in production it will make any difference at all. In the end it is not the poll/epoll overhead that determines your overall throughput. If you call poll/epoll more frequently than you should then it starts to add up but in reality one call per several thousand file system operations doing real IO is not going to make a big difference. Of course all the little bits help and I'm happy to see someone…

I keep seeing this...misconception? I don't know what to call it, over most comments. The article doesn't talk about optimizations. It talks about design decisions . Someone actually took the time to sit down, ponder what kind of workloads will be handled by his application, came across an interesting dilemma, measured and tested them both, and finally reached an educated conclusion. That right there is how you corre…

what good thinking gives you when your assumptions are wrong?

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#124

Post by pmjordan » Wed, Aug 04, 2010, 7:40 AM UTC

rbanffy wrote: post #92

pmjordan wrote: post #81

Earlier quoted context omitted.

Will you maintain Mongrel forever? With all due respect, this seems a ridiculous question. Do you obtain written statements to the above effect from all maintainers of software (open source or otherwise) before using it? Yes, it'll have to be maintained, but Zed's article alone is more documentation than you could ever hope for from most programmers. It's clear he isn't "most programmers" but that's no reason to hold…

I am not holding him to ridiculous standards. It's obvious he won't maintain Mongrel forever. What I am confronting him is with the fact that he may be able to navigate any arbitrarily clever construct he invents, but that future maintainers who inherit Mongrel may not be as capable. Again, it's his project and he's free to do whatever he feels like with it. Heck... I am not even a user. I offered him advice he is fr…

While we're at it, let's lobby our governments to ban all software innovation!

Seriously, I bet the operating system you're using relies on far more of that evil "cleverness" than the subject of this thread.

This whole thread reeks of thinly veiled ad hominem attacks to me.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#125

Post by FooBarWidget » Wed, Aug 04, 2010, 9:20 AM UTC

kqueue wrote: post #122

FooBarWidget wrote: post #108

Earlier quoted context omitted.

This sounds really strange to me. The kernel has full access to the page tables so can't it lookup things in userspace?

When the kernel is executing a function call placed on the stack, all the addresses on the stack are assumed in the same vm space. It does not know that an address is actually a virtual memory address belonging to process X and tries to figure out the value in the physical memory.

Yeah but it's possible to look up things in userspace right? So just change poll() to assume that the pointer points to userspace. I don't see the need for copying.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#126

Post by jacquesm » Wed, Aug 04, 2010, 10:22 AM UTC

starkfist wrote: post #114

jacquesm wrote: post #113

Earlier quoted context omitted.

> I'm a scientist I've never come across a scientist that took criticism of their work the way you do and that responded in the way you do. Shouting down, deriding, insulting and in general being a jerk to those that don't agree with you because 'you're a scientist' is not the way of science.

I dunno, that is how a lot of scientists I know interact with people. :/

That's pretty sad.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#127

Post by rbanffy » Wed, Aug 04, 2010, 11:44 AM UTC

pmjordan wrote: post #124

rbanffy wrote: post #92

Earlier quoted context omitted.

I am not holding him to ridiculous standards. It's obvious he won't maintain Mongrel forever. What I am confronting him is with the fact that he may be able to navigate any arbitrarily clever construct he invents, but that future maintainers who inherit Mongrel may not be as capable. Again, it's his project and he's free to do whatever he feels like with it. Heck... I am not even a user. I offered him advice he is fr…

While we're at it, let's lobby our governments to ban all software innovation! Seriously, I bet the operating system you're using relies on far more of that evil "cleverness" than the subject of this thread. This whole thread reeks of thinly veiled ad hominem attacks to me.

Zed tends to do the ad-hominem part very well without any need for external help. And you are using a straw-man. I never came even close to advocate for banning all software innovation.

What I said, and repeat, is that if you want to introduce an expensive to maintain piece of code, you have to weight the added cost against the performance gain. In this case, the performance gain seems marginal, the assumption of usage envisioned seems wrong and the added complexity seems just pointless.

It's his project, his code and I am not even a Mongrel user. I offer this as a friendly piece of advice, from old programmer to young programmer.

You know: it doesn't matter if you are beam-racing a 6502 or writing networking code to run on 64-bit deeply pipelined processors, there are things that remain true. This is one of them.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#128

Post by frognibble » Wed, Aug 04, 2010, 1:59 PM UTC

zedshaw wrote: post #70

frognibble wrote: post #27

The blog post does not say if the epoll code uses level triggering or edge triggering. It would be interesting to see the results for both modes. The smaller number of system calls required for edge triggering might make a difference in performance.

That's entirely possible, but then you pay a penalty in complexity because you have to keep track of missed events yourself. I think (unproven) that it's actually a wash because of this.

At most, you need to track a couple of booleans per socket, one for read and one for write.

Depending on what you are doing, you might not even need to track these booleans. For example, on the read side you can ignore read events when you are not interested in reading. When you switch back to read interest, you can read the socket to see if data arrived while you ignored events. A similar strategy can be used on the write side.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#129

Post by zedshaw » Wed, Aug 04, 2010, 3:14 PM UTC

blasdel wrote: post #109

zedshaw wrote: post #94

Earlier quoted context omitted.

Oh, you mean do what I'm already doing? Measuring and developing ideas then testing them? It helps if you're going to comment that you actually read the words I use, not the ones you have in your head that make you sound like you're super smart.

Yes, you measured the ideal ATR inflection point for poll vs. epoll in your synthetic microbenchmark. But you guessed wildly about what ATRs people see in the real world: http://news.ycombinator.com/item?id=1572292 http://news.ycombinator.com/item?id=1572418

Yeah, 'cause there's no way I'll be able to test a real web server that I actually wrote based on this small test. This is a small test to test one specific thing, doing more would confound the test. Confounding. Look it up.

Incidentally, this is the same test everyone else uses, so if you thought it was bullshit why did you support it when people testing epoll with it were using it? Oh, because they used it to confirm your bias rather than disagree with it.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#130

Post by zedshaw » Wed, Aug 04, 2010, 3:17 PM UTC

jacquesm wrote: post #107

zedshaw wrote: post #98

Earlier quoted context omitted.

Wow here we are again, you not reading my article. I ran the same test that everyone else runs for poll vs. epoll, then used R to craft graphs and tested hypothesis. It was not a localhost test. So far all you've got is trolling HN comments. YOU WIN!

Pipes, localhost, who is counting, as far as I'm concerned that's the same thing, making it seem as if for the purpose of this test that's a significant difference is simply conversational trickery. If you have tested this on real live servers then there is no evidence of that in your posting, and to suggest that this: http://dpaste.de/32o8/ is anything but a localhost test is simply bogus. The only use case where yo…

Testing this on real live servers is confounding. Man you guys really don't get this. If you want to test epoll and poll over file descriptors you test that. You don't test a billion other things in a network server. That confounds your results.

But what's really amazing is this is the test the proponents epoll have been using for 8 years. Where was your objection back when they were using it for that?