I feel silly asking, but who or what is 'rtm'?
Some punk skript kiddie who broke the internet in 1988.
Why HN was slow and how Rtm fixed it
71–80 of 202 posts
Re: Why HN was slow and how Rtm fixed it
#72Since no one has mentioned it yet - Varnish-cache.org, written by a FreeBSD kernel hacker, has a very nice feature, in that it will put all overlapping concurrent requests for the same cacheable resource "on hold", only fetch that resource once from the backend, then serve the same copy to all. Nearly all the expensive content on HN would be cacheable by varnish. Then you can get it down to pretty close to "1 backend request per content change" and stop worrying about how arbitrarily slow the actual backend server is, how many threads, how you deal with the socket, garbage collection, and all that.
Re: Why HN was slow and how Rtm fixed it
#73pg: you could probably try to write an event driven HTTP server on top of Arc, so that you don't have this kind of problems. Something like node.arc
Also if I understand correctly you use flat files that are loaded into memory at startup. It seems like that switching to Redis could be an interesting idea in theory, as it is more or less the implementation of this concept in an efficient and networked way.
Probably with such changes you can go from 20 to a few hundreds requests per second without problems.
Re: Why HN was slow and how Rtm fixed it
#74Or you can put a reverse proxy in front.
http://joshua.schachter.org/2008/01/proxy.html
(Like I suggested in 2009...)
Re: Why HN was slow and how Rtm fixed it
#75Lisp is definitely not a slow language: you can handle the crazy rate of 20 requests/second on a multi-core server!
They are only using one core. See http://news.ycombinator.com/item?id=2116987 for verification.
Ooops, yes, you are right, sorry. Then I guess it's a pretty decent rate for a Core2 duo.
Re: Why HN was slow and how Rtm fixed it
#76Since no one has mentioned it yet - Varnish-cache.org, written by a FreeBSD kernel hacker, has a very nice feature, in that it will put all overlapping concurrent requests for the same cacheable resource "on hold", only fetch that resource once from the backend, then serve the same copy to all. Nearly all the expensive content on HN would be cacheable by varnish. Then you can get it down to pretty close to "1 backend…
You clearly don't understand the problem. Even mod_pagespeed or memcached would be more appropriate here: They are rate-limited by the LISP kernel anyway (we are talking about dynamic content here).
Re: Why HN was slow and how Rtm fixed it
#771 thread per connection??? Not doing continual GC in a separate thread and instead taking 7 seconds and blocking everything? What is this the 1990s?
Feel free to fork MzScheme and replace the garbage collector with a new one that runs continuously.
Re: Why HN was slow and how Rtm fixed it
#78Since no one has mentioned it yet - Varnish-cache.org, written by a FreeBSD kernel hacker, has a very nice feature, in that it will put all overlapping concurrent requests for the same cacheable resource "on hold", only fetch that resource once from the backend, then serve the same copy to all. Nearly all the expensive content on HN would be cacheable by varnish. Then you can get it down to pretty close to "1 backend…
Can't you only use varnish for mostly non-dynamic content? Like for example, wouldn't the fact that it displays my username and karma score at the top of the page make it so that you couldn't use varnish (or at least make it more difficult)?
Re: Why HN was slow and how Rtm fixed it
#79Re: Why HN was slow and how Rtm fixed it
#80Who's Filo? David Filo?
Yahoo's entire BSD kernel team report to filo still.
Last time I had lunch with him we talked about the minutia of DNS server implementations because I was working on some optimization tricks and he was really interested in seeing them get implemented.
filo is a really amazing guy. He's the most down to earth billionaire I know. He talks way more about his family and hacking more than 'stuff'. If he isn't hacking on code as much as he used to it's because he cares about his company and is doing an important job looking after technical stuff that needs doing, even if it isn't interesting.