Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why HN was slow and how Rtm fixed it

ycombinator.com

Post reply on HN
Print view View original

61–70 of 202 posts

Re: Why HN was slow and how Rtm fixed it

#61

Post by wmf »

kujawa wrote: post #33
jey wrote: post #3

> when [MzScheme] wants to find a thread to run, it asks the O/S kernel about each socket in turn to see if any input is ready They've never heard of select()? But really, is there some reason that it's hard to collect up all the fds at once or something?

Read C10K? Both select() and poll() have this problem internally. You have to use one of the more advanced techniques available if you really want to scale. epoll(), kqueue() or friends.

Based on RTM's description, it sounds like even select() would be a huge improvement. HN needs to solve the C100 problem before worrying about C10K.

Re: Why HN was slow and how Rtm fixed it

#62

Post by swannodette »

jrockway wrote: post #46
svlla wrote: post #31

Earlier quoted context omitted.

not to mention that one thread per connection is, well, extremely outdated.

I don't know much about MzScheme, but it's quite possible that "thread" means "stack", not "OS thread". One context stack per TCP connection is quite sustainable; with Haskell's threads and Perl's coros, I run out of fds long before I'm using any significant amount of memory. (This is somewhere around 30,000 open connections on my un-tweaked Linux desktop. I know I can do a lot more if I tried.) The issue, in the cas…

My understanding is that MzScheme / Racket has a proper event loop.

Re: Why HN was slow and how Rtm fixed it

#63

Post by SageRaven »

dauphin wrote: post #30

> It turns out there is a hack in FreeBSD, invented by Filo, that causes the O/S not to give a new connection to the server until an entire HTTP request has arrived. This might reduce the number of threads a lot, and thus improve performance; I'll give it a try today or tomorrow. Anyone know if they're referring to "accept filters" here? FreeBSD folks can "man accf_http" if they're curious, which does prevent a reque…

Anyone know if they're referring to "accept filters" here? FreeBSD folks can "man accf_http" if they're curious, which does prevent a request from being handed off to the application until the complete (and valid?) request has been made. Certainly not a "hack" but a feature of the OS itself.

Re: Why HN was slow and how Rtm fixed it

#64

Post by kirubakaran »

djcapelis wrote: post #56
ezalor wrote: post #19

Reverse-proxying via nginx would solve this problem and more: the arbitrary 30 second limit on form submission (hotspots sometimes are slow...), nginx could handle rate limiting & logging instead of srv.arc, etc. The Arc codebase would btw be smaller and cleaner (no policy/sanitization code, etc.). Serving static content via Apache was a first step ;-) Don't reinvent the wheel!

Did you just tell people running a company that reinvented funding with their custom written news site written in their own programming language with a custom web server that they shouldn't re-invent the wheel? They think they can build a better wheel. They seem to like doing it and have a habit of it. There's nothing wrong with that.

Unless they built their server with nand gates, I don't see running nginx reverse proxy to be incoherent with their philosophy.

Re: Why HN was slow and how Rtm fixed it

#65

Post by idoh »

klochner wrote: post #38
blinkingled wrote: post #13

Sounds like there is a scalability issue within MzScheme in that it iterates over the number of threads, asking each thread about the sockets it has. As one can tell, once # of threads and # of sockets grow - finding which thread to run in user space becomes awfully expensive. As any clever admin will do, a least invasive fix involving limiting the number of connections and threads was done - with what sounds like im…

I think this whitepaper covers the plt web server bundled with MzScheme (now 'Racket'): http://www.cs.brown.edu/~sk/Publications/Papers/Published/kh...

Hacker News uses a web server written in arc.

Re: Why HN was slow and how Rtm fixed it

#69

Post by jcapote »

aristus wrote: post #15
gms wrote: post #4

Who's Filo? David Filo?

Yep. He's still a mensch. Yahoo did some incredible stuff on Apache & FreeBSD back in the day. I remember a hack that added hardcoded HTTP headers to the image files on disk, to squeeze that extra nilth percent out of the server.

That's awesome!

Re: Why HN was slow and how Rtm fixed it

#70

Post by tmsh »

jey wrote: post #3

> when [MzScheme] wants to find a thread to run, it asks the O/S kernel about each socket in turn to see if any input is ready They've never heard of select()? But really, is there some reason that it's hard to collect up all the fds at once or something?

I imagine mzscheme is using select under the hood. But I'm surprised they're not using poll. There's something up with the racket page, so I'm not able to download the mzscheme source just now (via http://arclanguage.org/install), but that's one thing I've noticed in Python web server implementations. Side-stepping epoll, libevent and all that for a second, there's a tendency to use select (which uses an array of file descriptors; 0 to highest file descriptor) instead of poll (which isn't encumbered the same way).

I think in part there is a tendency, among server developers, correctly, to fear anything that looks like a busy wait (e.g., with the name poll). But really poll is just as asynchronous as select in this context (I don't know about FreeBSD's implementation -- but Linux puts to sleep wait queues the same way, afaik). It just doesn't suffer from the crazy indexing scheme of select....

At any rate, I didn't get a chance to finish probing the internals of what mzscheme uses. But if there's a way to substitute poll for select, it can often alleviate those issues of 900 requests queue up and you eventually have an fd with a value of 1024 or greater -- even though you may not have 1024 actual concurrent requests....

Though others feel free to correct me if I'm wrong. I only comment because I came across a similar issue recently. This link may be useful too:

http://www.makelinux.net/ldd3/chp-6-sect-3.shtml

ETA. i finally got a copy of the most recent racket source (though probably not the one rtm and pg are using). but if anyone is curious, browse racket/src/network.c. the source version for mac uses a bunch of selects (e.g., for tcp-accept). replacing with poll might help.... the max number of FDs per login session is often 1024 by default. so you might want to bump that up if it's not already. and consider using poll.... just an idea.

Post reply on HN