Zed Shaw: "poll, epoll, science, and superpoll" with R • Hacker News Board

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#61

Post by zedshaw » Tue, Aug 03, 2010, 7:24 PM UTC

FooBarWidget wrote: post #6

pmjordan wrote: post #4

Pardon my ignorance, I haven't built high performance servers at this low a level, but I'm intrigued: What exactly is the definition of an "active" file descriptor in this context? My best guess after reading the man pages is that poll() takes an array of file descriptors to monitor and sets flags in the relevant array entries, which your code then needs to scan linearly for changes, whereas epoll_wait() gives you an…

An active file descriptor is one that you can read from or write to without blocking or getting EAGAIN as error. The whole point of poll/epoll/kqueue/select is to figure out which file descriptors are in such a state. The difference between poll and epoll is that, given an input of N file descriptors, poll returns all N file descriptors and you need to loop through each one of them to check whether the 'active' flag…

I totally absolutely agree they should fix epoll, but the way they've designed I don't see it happening. Of course they could fix the call for doing the actual select and make it at least as fast as poll, but the fact that you have to do a syscall for every file descriptor is idiotic.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#62

Post by zedshaw » Tue, Aug 03, 2010, 7:33 PM UTC

neilc wrote: post #59

zedshaw wrote: post #57

Earlier quoted context omitted.

> And as jacquesm points out, in a web-facing server, that's the case you should care about. Yes, but where's the evidence what people see for active/total ratios in the real world? I'm showing that unless it's below about 60% (probably more like 50%) then poll is the way to go. 60% active isn't entirely unrealistic at all. I can see quite a few servers hitting those thresholds, so in that cases, poll vs. epoll doesn…

It's entirely possible that you have servers that are at 80-90% ATR all the time I'd be curious if you have any evidence that this occurs in practice. Even a busy server with clients of uniform + low latency, intuitively I'd expect fairly low ATRs. I think what's more important in what I'm finding is that you really need both. I'm not sure you do: the performance advantage of poll seems marginal at best. When ATR is…

This is the point where talking about it does nothing. Go measure it like I have. In fact, I'll give you your hypothesis to test:

"There are no servers that have an ATR of > 80%."

That's easy to test, and I'm damn positive you could find some that disprove your assertion.

More importantly though, you have this assertion:

"Using both poll and epoll has no advantage in performance."

Again, who knows, that's why I'm testing and trying out. That's the science part, since I've got no idea, but I'll give it a shot. And now that I've done an analysis that tells me what really matters, I'll be able to do very good tests for the different kinds of loads.

Incidentally, when people run performance tests against web servers to see how fast they serve files they're testing the server with an ATR at around 100%. Food for thought.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#63

Post by zedshaw » Tue, Aug 03, 2010, 7:37 PM UTC

jacquesm wrote: post #11

pmjordan wrote: post #7

Earlier quoted context omitted.

Thanks for the detailed explanation. Sounds like I was at least on the right track. But if you ask me, the real solution is to have the kernel team fix their epoll implementation performance issues instead of forcing people to work around it with hybrid approaches. That does indeed sound like a better conclusion. Other than the stupid single-syscall-per-fd requirement, there's nothing in epoll's interface that would…

I highly doubt that in production it will make any difference at all. In the end it is not the poll/epoll overhead that determines your overall throughput. If you call poll/epoll more frequently than you should then it starts to add up but in reality one call per several thousand file system operations doing real IO is not going to make a big difference. Of course all the little bits help and I'm happy to see someone…

You again? Man you got some beef.

Anyway, we do have something working. You can deploy Mongrel2 right now, and I do all the time. You can actually measure how fast it is, although that's going to be slow since we haven't done much tuning so kind of pointless.

So, again you're wrong. We are at a point in the design where this measurement and analysis matters, and we have something that actually works to put it in. You should probably maybe go do some actual reading instead of posting here like you know what you're talking about.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#64

Post by zedshaw » Tue, Aug 03, 2010, 7:46 PM UTC

jacquesm wrote: post #13

jakevoytko wrote: post #12

Earlier quoted context omitted.

A conclusion reached by measurement is not premature. This looks like an attempt to write a better server than the 80/20 rule allows. If he's wrong and only one polling method is useful in production, the live servers will pick the good one and nobody will suffer because he jumped to conclusions. Since he's written Mongrel, I trust that he has a reason to worry about polling that may not have appeared in the post

> A conclusion reached by measurement is not premature. That's just plain wrong. Premature optimisation does not refer to having to measure before you optimise, it refers to optimising things that in practice may have little or no effect on the actual performance of the program. By doing these tests in isolation instead of while running on a profiling kernel under production load it is very well possible that the bot…

Again with this idea that Mongrel2 isn't working. You sir have no freaking idea what you're talking about.

"That's just plain wrong. Premature optimisation does not refer to having to measure before you optimise, it refers to optimising things that in practice may have little or no effect on the actual performance of the program."

No, that's just plain wrong. Premature optimisation is actually implementing something convoluted thinking it's optimized without knowing whether it actually is or not. It's voodoo cargo cult science. It's going against occam's razor.

There's nothing in Mongrel2 that's premature optimized. It's all very simple algorithms chosen for the right task, and later on I'll be testing them to see if they're still right. So your claim that this is premature optimization is just a buzzword and completely offensive. I took a long time to actually test my ideas before implementing them.

That's the total inverse of premature optimization.

What is total voodoo junk science is most of what you say. So far I haven't seen one set of data or any scientific experimental design or even a single testable hypothesis backing what you claim. It's all just rhetoric.

Until you've got hard numbers backing what you say, everything you're saying is inferior to what I'm doing: science.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#65

Post by zedshaw » Tue, Aug 03, 2010, 7:49 PM UTC

jacquesm wrote: post #22

jakevoytko wrote: post #20

Earlier quoted context omitted.

"I need polling" => "Here are my options, which one is better?" => "They're good for different things" => "I'll pick the best one for the environment" is a reasonable design process. More so than some decisions that I make! Yes, there's a fixed time budget. But you're suggesting selectively ignoring evidence when designing a program, preferring random guessing and pattern matching to actual numbers. Should he have co…

The funny thing is that he's either going to pick the wrong solution for the workload or spend a lot of time on creating a hybrid which will work as good as epoll would without augmentation. Zed is clearly out to change the world and I would very much like him to succeed but he just seems to be missing the obvious here, which is that idle connections are the ones to worry about (because they're very expensive!) so hi…

First off, again you miss the point that I've already got poll working in Mongrel2. If this fails (which I'll know since I like, measure stuff rather than care a bottle of cheetah blood around like you) then I'll just go put epoll in there or leave it with poll.

But hey, if I don't make it back alive from my complete dangerous experiment in disproving that epoll is always the way to go always you can come get me. Bring a big gun because this stuff is so scawy and howwible I might not make out alive.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#66

Post by zedshaw » Tue, Aug 03, 2010, 7:52 PM UTC

llimllib wrote: post #42

jakevoytko wrote: post #20

Earlier quoted context omitted.

"I need polling" => "Here are my options, which one is better?" => "They're good for different things" => "I'll pick the best one for the environment" is a reasonable design process. More so than some decisions that I make! Yes, there's a fixed time budget. But you're suggesting selectively ignoring evidence when designing a program, preferring random guessing and pattern matching to actual numbers. Should he have co…

> => "I'll pick the best one for the environment" Which, in reality is, "I'll spend a lot of design and implementation effort designing a new one which may or may not improve the measurable, global performance of my new web server because it's not yet at the point where I can benchmark these sorts of things to verify that I'm not wasting a whole ton of effort that could be better spent by deciding that epoll is fast…

It's an idea I had after actually measuring. If it doesn't work then I tried something out.

What you really should be getting from it though is that epoll is not faster. It is not O(1). It is not faster on smaller vs. larger lists of FDs. Pretty much all the things you were told as advantages of epoll are total crap.

The only advantage of epoll is it's O(N=active) when poll is O(N=total). That's it.

So at a minimum I've done some education and spent some time learning something.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#67

Post by zedshaw » Tue, Aug 03, 2010, 7:53 PM UTC

axod wrote: post #21

jakevoytko wrote: post #12

Earlier quoted context omitted.

A conclusion reached by measurement is not premature. This looks like an attempt to write a better server than the 80/20 rule allows. If he's wrong and only one polling method is useful in production, the live servers will pick the good one and nobody will suffer because he jumped to conclusions. Since he's written Mongrel, I trust that he has a reason to worry about polling that may not have appeared in the post

FWIW, I've written an async webserver which handles a few thousand concurrent users, and does thousands of HTTP reqs/second. I've never seen poll/epoll as a bottleneck, but I'm using Java NIO which really seems to work extremely well (I don't remember how/which it uses, I think epoll). Maybe Java does some of this cool stuff already so perhaps I'm shielded from the pain of dealing with things directly. In the past I'…

> But I'm skeptical those numbers are true. That was my point.

Yeah, I'm skeptical those numbers are true too, but then again we're talking about totally different numbers.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#68

Post by zedshaw » Tue, Aug 03, 2010, 7:56 PM UTC

rbanffy wrote: post #29

pmjordan wrote: post #25

Earlier quoted context omitted.

I haven't actually used Mongrel2 yet, but I get the impression you can use it as a front-end for multiple physical servers, in which case the relative load of the M2 process on its server increases. Besides, unless Zed is writing your application and this is eating away at his time for that, it's not clear what exactly your complaint is.

> it's not clear what exactly your complaint is. Any complexity introduced in the code increases its long term maintenance cost. I strongly suspect this is one of those cases where the performance gain will not justify the long-term effort of maintaining a more complex architecture. I remember having a similar discussion circa 1992 about advantages and disadvantages of using ODBC versus native MS SQL/Sybase libraries…

So, you're comparing my track record with writing simple maintainable well documented code to something you did in 1992 with ODBC? That's your experience that's causing all the paranoia?

The worst two things that afflict programmers today is:

1. They never update their information, even after 18 years (18! You realize that right!? Things change man!)

2. They have an irrational paranoia about trying new things, as if me trying this out is going to destroy the universe.

That really needs to change.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#69

Post by bdr » Tue, Aug 03, 2010, 7:58 PM UTC

zedshaw wrote: post #58

jacquesm wrote: post #2

In real-life web serving situations, and not in benchmarks, the majority of the fds is not active. It's the slow guys that kill you. A client on a fast connection will come in and will pull the data as fast as the server can spit it out, keeping the process and the buffers occupied for the minimum amount of wall clock time and the number of times the 'poll' cycle is done is very small. But the slowpokes, the ones on…

So let's take your assertions and take them apart: > the ones on dial up and on congested lines will get you every time. Do you have numbers on the dial-up users for your server? My understanding is that there's far fewer, so this is bogus. Show evidence of high dial-up penetration first. > They keep the processes busy far longer than you'd want and you have to hit the 'poll' cycle far more frequently Again, you have…

Read "on dial-up" as "slow". The argument depends only on there being a certain distribution of client speeds. It's not about dial-up in particular.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#70

Post by zedshaw » Tue, Aug 03, 2010, 7:59 PM UTC

frognibble wrote: post #27

The blog post does not say if the epoll code uses level triggering or edge triggering. It would be interesting to see the results for both modes. The smaller number of system calls required for edge triggering might make a difference in performance.

That's entirely possible, but then you pay a penalty in complexity because you have to keep track of missed events yourself. I think (unproven) that it's actually a wash because of this.