Not being able to handle 20 requests/sec quickly, in 2011, for a read-mostly website is just shameful.
Why HN was slow and how Rtm fixed it
151–160 of 202 posts
Re: Why HN was slow and how Rtm fixed it
#152Earlier quoted context omitted.
Right, they only re-invent wheels when they feel they can make a better one. I don't think they feel they can go through the effort of fabricating a better chip. Though I wouldn't put it past rtm to try. The philosophy "Don't reinvent the wheel" however, is definitely inconsistent with their philosophy. They will reinvent the wheel whenever they feel they can make a better one. Just because they haven't reinvented ev…
I think he means that the nginx reverse proxy is just a part of the infrastructure, like the server, OS, MzScheme etc they use.
Totally. "If you want to make an apple pie from scratch, you first need to recreate the Universe" -- Sagan.
Re: Why HN was slow and how Rtm fixed it
#153"In 7 seconds, a hundred or more connections accumulate. So the server ends up with hundreds of threads, most of them probably waiting for input (waiting for the HTTP request). MzScheme can be inefficient when there are 100s of threads waiting for input -- when it wants to find a thread to run, it asks the O/S kernel about each socket in turn to see if any input is ready, and that's a lot of asking per thread switch…
Or I don't know, use continuations in a place that's actually appropriate? John Fremlin showed that even with horrible CPS rewriting and epoll you can get way better throughput in SBCL (TPD2) than nginx. MzScheme comes with native continuations. It's not hard to call out to epoll. Instead everyone in the Lisp community (pg included) is still enamored with using continuations to produce ugly URLs and unmaintainable we…
MzScheme/Racket's continuations are of the "copy the C stack" variety, or were last time I checked. They are in no way efficient; it would probably be better to CPS transform your own code than try to use MzScheme/Racket's continuations directly in performance sensitive code.
Re: Why HN was slow and how Rtm fixed it
#154I find it disturbing to see people asking "Who is Rtm?" "Who is filo?" I understand if you are in tech you might not know figures in history or literature... but these guys? Every time you login to a UNIX/Linux system you use the passwd file and related setup - authored at least in part by Rtm's father. http://www.manpages.info/freebsd/passwd.1.html Rtm has done lots in his own right as the wikipdia pages show. But s…
You're making little sense. So "go learn" but "don't ask?"
You write for a major magazine on the subject. Yes - I would hope you know more about the history of this topic than I do.
Re: Why HN was slow and how Rtm fixed it
#155I find it disturbing to see people asking "Who is Rtm?" "Who is filo?" I understand if you are in tech you might not know figures in history or literature... but these guys? Every time you login to a UNIX/Linux system you use the passwd file and related setup - authored at least in part by Rtm's father. http://www.manpages.info/freebsd/passwd.1.html Rtm has done lots in his own right as the wikipdia pages show. But s…
This has to be one of the most unnecessary comments I have seen recently on HN. Civility people. What happened to that? Btw, I down vote me if you like, but it's true. It's easy for us to get caught up in our own brilliance that we talk down to others that don't know as much in a particular subject as we do. Ironically, it shows more about you, than it does them.
Civility? Go to a sports blog - put on you Bio that you are a journalist for sports illustrated and then in the commentary of an article on American football ask the question "who is Joe Namath?" Then judge my commentary by its relative civility.
Do you not read about the authors of books you read - music you listen to? Why then wouldn't you read up on the people whose software you use. I am sorry - it is ignorance and on HN that deserves to be called out.
Re: Why HN was slow and how Rtm fixed it
#156> when [MzScheme] wants to find a thread to run, it asks the O/S kernel about each socket in turn to see if any input is ready They've never heard of select()? But really, is there some reason that it's hard to collect up all the fds at once or something?
Read C10K? Both select() and poll() have this problem internally. You have to use one of the more advanced techniques available if you really want to scale. epoll(), kqueue() or friends.
poll() is slightly better than select(), because you only have to iterate over the file descriptors that were passed, rather than from 0 to nfds.
Re: Why HN was slow and how Rtm fixed it
#157Re: Why HN was slow and how Rtm fixed it
#158"In 7 seconds, a hundred or more connections accumulate. So the server ends up with hundreds of threads, most of them probably waiting for input (waiting for the HTTP request). MzScheme can be inefficient when there are 100s of threads waiting for input -- when it wants to find a thread to run, it asks the O/S kernel about each socket in turn to see if any input is ready, and that's a lot of asking per thread switch…
Can't upvote you enough for this well-written post!
Re: Why HN was slow and how Rtm fixed it
#159Earlier quoted context omitted.
Sounds terribly inefficient to me, but what do I know -shrug-
All control flow is a subset of continuations. The stack is a continuation (calling a function is call-with-current-continuation, return is just calling the "current continuation"), loops are continuations (with the non-local control flow, like break/last/redo/etc.), exceptions are continuations (like functions, but returning to the frame with the error handler), etc. Continuations are the general solution to things…
In theory yes, in practice you need to reify the stack (even for one-shot continuations). Clinger, Hartheimer and Ost have a really good survey paper of the different ways to do that:
http://www.scribd.com/doc/47221367/Clinger-Implementation-St...
Re: Why HN was slow and how Rtm fixed it
#160When I read this headline, my immediate thought was "Oh, he must have forgotten to shut down the copy of his worm that was running on the HN servers." http://en.wikipedia.org/wiki/Morris_worm
Please, retire that stupid "joke". It's 22 years too old.