Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why HN was slow and how Rtm fixed it

ycombinator.com

Post reply on HN
Print view View original

51–60 of 202 posts

Re: Why HN was slow and how Rtm fixed it

#51

Post by dauphin »

svlla wrote: post #37
dauphin wrote: post #30

> It turns out there is a hack in FreeBSD, invented by Filo, that causes the O/S not to give a new connection to the server until an entire HTTP request has arrived. This might reduce the number of threads a lot, and thus improve performance; I'll give it a try today or tomorrow. Anyone know if they're referring to "accept filters" here? FreeBSD folks can "man accf_http" if they're curious, which does prevent a reque…

this seems impossible for item pages due to how continuation ids are used for replies

This could be resolved using consistent hashing or a critbit tree.

Re: Why HN was slow and how Rtm fixed it

#54

Post by jwr »

Which is why it's good to have a mature VM underneath your language. Paul's choice of basing an implementation of Arc on MzScheme was a very good one (I remember people criticizing him for not building a standalone implementation with a new VM).

I write time-critical applications in Clojure and JVM's -XX:+UseConcMarkSweepGC flag is a lifesaver. We no longer get those multi-second pauses when full GC occurs.

Re: Why HN was slow and how Rtm fixed it

#55

Post by dauphin »

richcollins wrote: post #50

Sounds like a switch to async I/O would be helpful.

Well, HN is written in Arc, which is a layer on top of MzScheme. MzScheme handling of sockets is actually already done with the select() syscall, and its "threads" are lightweight non-blocking threads (think Erlang). So it's already async but with "sugar".

Re: Why HN was slow and how Rtm fixed it

#56

Post by djcapelis »

ezalor wrote: post #19

Reverse-proxying via nginx would solve this problem and more: the arbitrary 30 second limit on form submission (hotspots sometimes are slow...), nginx could handle rate limiting & logging instead of srv.arc, etc. The Arc codebase would btw be smaller and cleaner (no policy/sanitization code, etc.). Serving static content via Apache was a first step ;-) Don't reinvent the wheel!

Did you just tell people running a company that reinvented funding with their custom written news site written in their own programming language with a custom web server that they shouldn't re-invent the wheel?

They think they can build a better wheel. They seem to like doing it and have a habit of it. There's nothing wrong with that.

Re: Why HN was slow and how Rtm fixed it

#58

Post by jey »

kujawa wrote: post #33
jey wrote: post #3

> when [MzScheme] wants to find a thread to run, it asks the O/S kernel about each socket in turn to see if any input is ready They've never heard of select()? But really, is there some reason that it's hard to collect up all the fds at once or something?

Read C10K? Both select() and poll() have this problem internally. You have to use one of the more advanced techniques available if you really want to scale. epoll(), kqueue() or friends.

Rtm's phrasing implies that MzScheme is making a syscall for each socket in turn, so it sounds like it isn't even using the basic select() or poll().
Post reply on HN