Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zed Shaw: "poll, epoll, science, and superpoll" with R

sheddingbikes.com

Post reply on HN
Print view View original

81–90 of 149 posts

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#81

Post by pmjordan »

rbanffy wrote: post #76
zedshaw wrote: post #68

Earlier quoted context omitted.

So, you're comparing my track record with writing simple maintainable well documented code to something you did in 1992 with ODBC? That's your experience that's causing all the paranoia? The worst two things that afflict programmers today is: 1. They never update their information, even after 18 years (18! You realize that right!? Things change man!) 2. They have an irrational paranoia about trying new things, as if…

1) Will you maintain Mongrel forever? It's not your track record that's the question, but the one of all future maintainers of Mongrel that will have to deal with the added complexity this change creates. 2) The experience from 1992 still seems current. Adding complexity to any software project adds cost to maintain it in the future. My experience in 1992 showed how added complexity for a marginal performance gain di…

Will you maintain Mongrel forever?

With all due respect, this seems a ridiculous question. Do you obtain written statements to the above effect from all maintainers of software (open source or otherwise) before using it? Yes, it'll have to be maintained, but Zed's article alone is more documentation than you could ever hope for from most programmers. It's clear he isn't "most programmers" but that's no reason to hold him to ridiculous standards.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#82

Post by [deleted] »

rbanffy wrote: post #76
zedshaw wrote: post #68

Earlier quoted context omitted.

So, you're comparing my track record with writing simple maintainable well documented code to something you did in 1992 with ODBC? That's your experience that's causing all the paranoia? The worst two things that afflict programmers today is: 1. They never update their information, even after 18 years (18! You realize that right!? Things change man!) 2. They have an irrational paranoia about trying new things, as if…

1) Will you maintain Mongrel forever? It's not your track record that's the question, but the one of all future maintainers of Mongrel that will have to deal with the added complexity this change creates. 2) The experience from 1992 still seems current. Adding complexity to any software project adds cost to maintain it in the future. My experience in 1992 showed how added complexity for a marginal performance gain di…

[deleted]

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#83

Post by lukesandberg »

interesting article! Is 'super-poll' done yet? i would have liked to see a super poll line on some of those graphs to see how it compares to just vanilla poll and ePoll at different ATRs. Though i guess you would also have to test for situations where ATR varies over time (so that you could measure the impact of moving fds back and forth).

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#84

Post by jfager »

zedshaw wrote: post #57
jfager wrote: post #17

It is worth pointing out that the original epoll benchmarks were focused on how performance scaled with the number of dead connections, not performance in general: http://www.xmailserver.org/linux-patches/nio-improve.html And as jacquesm points out, in a web-facing server, that's the case you should care about. A 15-20% performance hit in a situation a web-facing server is never going to see doesn't matter when you c…

> And as jacquesm points out, in a web-facing server, that's the case you should care about. Yes, but where's the evidence what people see for active/total ratios in the real world? I'm showing that unless it's below about 60% (probably more like 50%) then poll is the way to go. 60% active isn't entirely unrealistic at all. I can see quite a few servers hitting those thresholds, so in that cases, poll vs. epoll doesn…

IE will keep a connection open for about 60 seconds, how much of that's going to be active? I don't have the exact number, and of course it will vary, but of course it's going to be far less than 60% in the vast majority of cases.

If a site gets spiked with the typical 'read-and-leave' traffic a link from reddit or huffpo or wherever generates, how does superpoll compare to straight epoll? Based on your description so far, I can only see it hurting - you're not just wasting time on dead connections in your poll bin, you're now also incurring the overhead of managing the migration over to the epoll bin.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#85

Post by rortian »

zedshaw wrote: post #74
bdr wrote: post #69

Earlier quoted context omitted.

Read "on dial-up" as "slow". The argument depends only on there being a certain distribution of client speeds. It's not about dial-up in particular.

And, if there's a distribution of speeds then you can measure the distribution and see what works best. Again, my challenge still stands: Measure it or STFU.

If only you practiced what you preached. Imagine the amount of self-righteous bile that your servers wouldn't serve.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#86

Post by blasdel »

zedshaw wrote: post #57
jfager wrote: post #17

It is worth pointing out that the original epoll benchmarks were focused on how performance scaled with the number of dead connections, not performance in general: http://www.xmailserver.org/linux-patches/nio-improve.html And as jacquesm points out, in a web-facing server, that's the case you should care about. A 15-20% performance hit in a situation a web-facing server is never going to see doesn't matter when you c…

> And as jacquesm points out, in a web-facing server, that's the case you should care about. Yes, but where's the evidence what people see for active/total ratios in the real world? I'm showing that unless it's below about 60% (probably more like 50%) then poll is the way to go. 60% active isn't entirely unrealistic at all. I can see quite a few servers hitting those thresholds, so in that cases, poll vs. epoll doesn…

> but where's the evidence what people see for active/total ratios in the real world?

Yes Zed, where the fuck is it? You're claiming SCIENCE! based on your worst-case synthetic localhost benchmarks, and then turning around and wildly guessing as to real-world performance characteristics with internet latencies.

Worse, your whole thesis hinges off of ATR but you made no effort to measure it anywhere, instead you're passive-aggressively berating us to do it.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#87

Post by jemfinch »

zedshaw wrote: post #58
jacquesm wrote: post #2

In real-life web serving situations, and not in benchmarks, the majority of the fds is not active. It's the slow guys that kill you. A client on a fast connection will come in and will pull the data as fast as the server can spit it out, keeping the process and the buffers occupied for the minimum amount of wall clock time and the number of times the 'poll' cycle is done is very small. But the slowpokes, the ones on…

So let's take your assertions and take them apart: > the ones on dial up and on congested lines will get you every time. Do you have numbers on the dial-up users for your server? My understanding is that there's far fewer, so this is bogus. Show evidence of high dial-up penetration first. > They keep the processes busy far longer than you'd want and you have to hit the 'poll' cycle far more frequently Again, you have…

> Do you have numbers on the dial-up users for your server? My understanding is that there's far fewer, so this is bogus. Show evidence of high dial-up penetration first.

He doesn't need to show that it's high, only that it's high enough to cause a significant contingent of ordinary webservers' requests to be lingering slow connections.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#88

Post by kqueue »

FooBarWidget wrote: post #78
kqueue wrote: post #54

Earlier quoted context omitted.

For the kernel to execute a system call, it has to place the arguments on its stack. a system call doesn't execute in the userland.

Yes but the argument to poll is a pointer. The pointer would be copied but the kernel can still follow the pointer to userspace, right?

The pointer referred to by the process is not accessible by the kernel because when the user process was running, it had a different vm space than the kernel vm space. So if it just passes the pointer (without copying the pointer's data), then the kernel will point to a virtual address that won't exist until the user process gets swapped in again.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#89

Post by blasdel »

zedshaw wrote: post #74
bdr wrote: post #69

Earlier quoted context omitted.

Read "on dial-up" as "slow". The argument depends only on there being a certain distribution of client speeds. It's not about dial-up in particular.

And, if there's a distribution of speeds then you can measure the distribution and see what works best. Again, my challenge still stands: Measure it or STFU.

How about you get measurements of ATR from real-world deployments instead of the wild conjectures you've laid in this thread? Your challenge applies even moreso to yourself:

Measure it or GTFO.

Re: Zed Shaw: "poll, epoll, science, and superpoll" with R

#90

Post by mhd »

aston wrote: post #80
zedshaw wrote: post #73

Earlier quoted context omitted.

I was hoping for e, but alas no luck.

It's pretty darn close to 1 - 1/e.

The first four digits of 1/0.6 would be 1666, the Annus Mirabilis. So you could compare Mongrel2's multiple request handlers to Isaac Newton first splitting light with a prism.
Post reply on HN