Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zed Shaw: "poll, epoll, science, and superpoll" with R

sheddingbikes.com

Post reply on HN
Print view View original

111–120 of 149 posts

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#111

Post by jacquesm »

blueben wrote: post #106
jacquesm wrote: post #75

Earlier quoted context omitted.

So, if it's working why not throw a load of real world traffic at it in stead of this 'science' that you're performing here ? After all, that is where the rubber meets the road and it would be a very easy way to determine if your hunch is right or not. Epoll was specifically created with that sort of workloads in mind, your 'surprising' conclusion is not rooted in the fact that epoll is somehow behaving in a way that…

Because "real world traffic" is a bullshit test. Who's real world traffic? Mine? Yours? Google's? Yahoo's? Science is repeatable, which means you can replicate the same inputs to the test every time. The lab experiments involved in science are often highly sanitized and not "real world" at all. Determining what the results mean in the real world comes later. If you're going to complain about science, at least underst…

There are basically three sorts of traffic that I have experience with and would expect to be the major portion of whatever goes over the web:

- underwater ajax requests

- regular website content (images, dynamic html, css and other relatively small (say - media servers (filedumps, video servers, streaming audio servers)

Each of those requires fairly specific tuning of the TCP stack to get the most out of it, so you're not likely going to find all of these on one and the same machine unless it is a small operation (and in that case this whole discussion is moot).

A benchmark done in isolation is meaningless because in the end, real world traffic is what it is all about. So, I personally don't care whose site(s) you test with, as long as there are enough of them to get a statistically valid result.

Google's or Yahoo's would be fine with me, I've given my results above, if I have the time I'll do the same thing on a couple of other high volume sites.

I've (unfortunately) studied this problem quite a bit because of the size of the websites that I'm involved with and so far I've learned that you can play around on your testbench all day long it doesn't matter one little bit for production purposes unless you are very careful (such as in that other test linked from this page) to simulate users clients.

You could do a lot worse than to play back a log file in order to make an experiment repeatable. I assume that real world performance is what Zed is after, not theoretical performance.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#112

Post by pphaneuf »

zedshaw wrote: post #64
jacquesm wrote: post #13

Earlier quoted context omitted.

> A conclusion reached by measurement is not premature. That's just plain wrong. Premature optimisation does not refer to having to measure before you optimise, it refers to optimising things that in practice may have little or no effect on the actual performance of the program. By doing these tests in isolation instead of while running on a profiling kernel under production load it is very well possible that the bot…

Again with this idea that Mongrel2 isn't working. You sir have no freaking idea what you're talking about. "That's just plain wrong. Premature optimisation does not refer to having to measure before you optimise, it refers to optimising things that in practice may have little or no effect on the actual performance of the program." No, that's just plain wrong. Premature optimisation is actually implementing something…

I think what he's saying is that it should be pretty easy to change between poll, epoll, and "superpoll", as you can easily abstract a common interface. Then, you could worry about the performance of that particular bit only when you encounter it, and use your time arguably more effectively on actual feature needed to productionize Mongrel2 better.

I sort of agree with him, except with an important detail: this question is basically about prioritization of your time, and I'd say that this is nobody's business but yours! You can optimize memcpy() all day, for all I care. ;-)

There's one aspect where you'd be quite right to do some investigation before implementing: if the outcome changes the interface you'd need to implement.

For example, here's an hypothesis: using epoll's edge-triggered mode could drastically reduce the number of events (since you only get an event when an fd becomes readable/writable the first time, instead of every time it's in that state). Since epoll is O(N) on the number of events returned (not on the number of fds that are currently readable/writable), you'd lower the effective ATR a whole lot. In fact, a really busy server would have fewer events, since a readable fd would stay that way for longer if data is received at a great rate (the write-side story might be less brilliant). You'd also have to do much fewer calls to epoll_ctl, since you could just stop caring about the reading side while you're trying to write the last batch of data on the other side (no need to remove it from the interest set, you won't get events for it). You only need to set it when flipping from read to write, and the other way (after receiving/sending headers and bodies).

Now, if that's true, that's a big deal, because now you have to change your design a fair bit. You have to remember that an fd is readable until you get EAGAIN from read() yourself, so there's some more state management, moving that fd from one list to another, etc. Finding out that this would be a million times faster (or slower!) now would save you a ton of work, either way.

But finding whether poll or epoll is faster, or an hybrid solution with the same interface? Meh, it could wait.

(about my hypothesis, that's actually what Davide Libenzi designed epoll for, which might explain some of the weirder bits)

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#113

Post by jacquesm »

zedshaw wrote: post #96
jemfinch wrote: post #87

Earlier quoted context omitted.

> Do you have numbers on the dial-up users for your server? My understanding is that there's far fewer, so this is bogus. Show evidence of high dial-up penetration first. He doesn't need to show that it's high , only that it's high enough to cause a significant contingent of ordinary webservers' requests to be lingering slow connections.

I agree, but "high enough" is apparently just 60%. The standing question is, what's the actual level in different kinds of servers? In other words, I've given a metsric, ATR at 60% is the break even point for poll vs. epoll. So far the only responses I've got haven't even tried to give out a metric, let alone say what their actual ATR is but they claim that it's low. I'm a scientist, so in the same way I don't believ…

> I'm a scientist

I've never come across a scientist that took criticism of their work the way you do and that responded in the way you do. Shouting down, deriding, insulting and in general being a jerk to those that don't agree with you because 'you're a scientist' is not the way of science.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#114

Post by starkfist »

jacquesm wrote: post #113
zedshaw wrote: post #96

Earlier quoted context omitted.

I agree, but "high enough" is apparently just 60%. The standing question is, what's the actual level in different kinds of servers? In other words, I've given a metsric, ATR at 60% is the break even point for poll vs. epoll. So far the only responses I've got haven't even tried to give out a metric, let alone say what their actual ATR is but they claim that it's low. I'm a scientist, so in the same way I don't believ…

> I'm a scientist I've never come across a scientist that took criticism of their work the way you do and that responded in the way you do. Shouting down, deriding, insulting and in general being a jerk to those that don't agree with you because 'you're a scientist' is not the way of science.

I dunno, that is how a lot of scientists I know interact with people. :/

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#115

Post by pphaneuf »

zedshaw wrote: post #73
jaekwon wrote: post #55

0.6 is so arbitrary. it should be 1.0/golden-ratio.

I was hoping for e, but alas no luck.

The best would be if it would be possible to code up superpoll to be adaptive, and in effect, benchmark itself to come to the same conclusion, dynamically. So if one day the kernel people fixed epoll to be better all the time, Mongrel2 would magically not use poll() much on systems using that kernel, and favour epoll.

Of course, that's often Kinda Hard To Do (tm). ;-)

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#116

Post by pphaneuf »

Question: as the ATR is going higher, so would the proportional time spent in poll or epoll, no?

So if you have a thousand fds, and they're all active, you have to deal with a thousand fds, which would make the difference between poll and epoll insignificant (only twice as fast, not even an order of magnitude!)?

This would make the micro-benchmark quite micro! Annoyingly enough, I think that means that the real way to find out would be an httpperf run, with each backends. A lot more work...

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#117

Post by jacquesm »

zedshaw wrote: post #100
gchpaco wrote: post #91

Earlier quoted context omitted.

You have measured nothing about real world workloads. You have applied completely artificial benchmarks and formed some possible conclusions from them that mean absolutely nothing until you demonstrate that this is a real problem in the real world. There's ample evidence to believe that FDs spend the majority of their existence idle--between HTTP keepalive, processing time for the queries themselves, network bandwidt…

I used the same test everyone has used for the last 8 years to compare poll vs. epoll performance. I also ran it in order to test a hypothesis then assumed I was wrong then ran it more and presented the information openly so others could try it. Of course I'm going to keep doing this, but if you say that my test is invalid, then all of the tests people did to justify epoll are invalid.

The tests are perfectly valid, it's the conclusion that is wrong. Of course there is a trade-off, of course you'll find a cross-over point because there is a trade off. The fact that the crossover point is at roughly 60% for your kernel config and machine setup is essentially meaningless, other than that it it at least falls within the expected range (50 to 80% or so).

Your hypothesis is that web servers have the majority of their fds active most of the time, and that's where the problem lies. I've put up the numbers elsewhere in this thread, feel free to do some measurement on your own high traffic websites to come up with more data points.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#119

Post by c00p3r »

It is a little wonder why this kind of people think that everyone else are just stupid to realize such things. What they want is a fame and followers. (btw, don't you forget to donate!)

hint: nginx/src/event/modules/ngx_epoll_module.c

May be one should learn how to use epoll and, perhaps, how to program? ^_^

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#120

Post by prodigal_erik »

mfukar wrote: post #37
jacquesm wrote: post #11

Earlier quoted context omitted.

I highly doubt that in production it will make any difference at all. In the end it is not the poll/epoll overhead that determines your overall throughput. If you call poll/epoll more frequently than you should then it starts to add up but in reality one call per several thousand file system operations doing real IO is not going to make a big difference. Of course all the little bits help and I'm happy to see someone…

I keep seeing this...misconception? I don't know what to call it, over most comments. The article doesn't talk about optimizations. It talks about design decisions . Someone actually took the time to sit down, ponder what kind of workloads will be handled by his application, came across an interesting dilemma, measured and tested them both, and finally reached an educated conclusion. That right there is how you corre…

Writing the "which fds are ready" check twice would be crazy, if neither version were markedly more efficient than the other. Optimization is the only reason to make this particular design decision, and measurement is definitely the right way to go about it.
Post reply on HN