Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why HN was slow and how Rtm fixed it

ycombinator.com

Post reply on HN
Print view View original

91–100 of 202 posts

Re: Why HN was slow and how Rtm fixed it

#91

Post by aonic »

nuclear_eclipse wrote: post #81
rarrrrrr wrote: post #72

Since no one has mentioned it yet - Varnish-cache.org, written by a FreeBSD kernel hacker, has a very nice feature, in that it will put all overlapping concurrent requests for the same cacheable resource "on hold", only fetch that resource once from the backend, then serve the same copy to all. Nearly all the expensive content on HN would be cacheable by varnish. Then you can get it down to pretty close to "1 backend…

Reverse proxies won't work for HN, because requests for the same resource from multiple users can't use the same results. Not only are certain bits of info customized for the user (like your name/link at the top), but even things like the comments and links are custom per user. Things like users' showdead value, as well as whether the user is deaded, can drastically change the output of each page. Eg, comments by a d…

Varnish supports edge side includes. The header bar could be an ESI and the rest of the page could be cached

Re: Why HN was slow and how Rtm fixed it

#92

Post by gills »

pg wrote: post #21
PStamatiou wrote: post #16

Earlier quoted context omitted.

flat files, no database

That's not the bottleneck. Essentially there's an in-memory database (known as hash tables). Stuff is lazily loaded off disk into memory, but most of the frequently needed stuff is loaded once at startup. The bottleneck is the amount of garbage created by generating pages. IIRC there is some horrible inefficiency involving UTF-8 characters.

Are you using any sort of in-memory fragment caching? That seems like it might reduce some render overhead.

Re: Why HN was slow and how Rtm fixed it

#93

Post by axod »

pg wrote: post #77
axod wrote: post #66

1 thread per connection??? Not doing continual GC in a separate thread and instead taking 7 seconds and blocking everything? What is this the 1990s?

Feel free to fork MzScheme and replace the garbage collector with a new one that runs continuously.

Surely there's a way to target the JVM?

I'd trust my life on the JVM, it's pretty battle tested, and the GC is simply awesome.

Re: Why HN was slow and how Rtm fixed it

#94

Post by ximeng »

nc17 wrote: post #8

YC ranks 2400 on Alexa, and I'm sure most of the traffic is HN. I bet you'd be hard-pressed to find a top 10k site written in Scheme. Does anyone know of one? http://www.alexa.com/siteinfo/ycombinator.com

And, presumably thanks to tptacek's hard work, one of the key search terms bringing people in here is "sous vide supreme".

Re: Why HN was slow and how Rtm fixed it

#95

Post by acangiano »

pg wrote: post #77
axod wrote: post #66

1 thread per connection??? Not doing continual GC in a separate thread and instead taking 7 seconds and blocking everything? What is this the 1990s?

Feel free to fork MzScheme and replace the garbage collector with a new one that runs continuously.

Do you ever question the choice of MzScheme?

Re: Why HN was slow and how Rtm fixed it

#96

Post by yummyfajitas »

dminor wrote: post #84
j_baker wrote: post #78

Earlier quoted context omitted.

Can't you only use varnish for mostly non-dynamic content? Like for example, wouldn't the fact that it displays my username and karma score at the top of the page make it so that you couldn't use varnish (or at least make it more difficult)?

Yes, you'd have to somehow separate the dynamic content from the static content so they could be fetched in different requests and then combined (probably via ajax). If it's just your username and karma then it's simple enough, but if the comments are displayed differently for different people then it could be bear.

Comments are displayed differently for different people.

I once posted a comment which was immediately invisible to everyone besides me - I'm guessing it was marked as spam for some reason, but left visible to me so I think I successfully posted it.

Re: Why HN was slow and how Rtm fixed it

#97

Post by pclark »

kabdib wrote: post #6

I worked at a startup once that made a network card that did this type of buffering (wait for whole HTTP requests, then forward as a lump to the host, across a local fast bus). Pretty whizzy, definitely helped server scaling. We started shipping in 2001; the dot-com bust more or less canceled any interest in the product, and canceled the company, too . . .

What was the name of the company/product?

Re: Why HN was slow and how Rtm fixed it

#98

Post by pig »

djcapelis wrote: post #83
kirubakaran wrote: post #64

Earlier quoted context omitted.

Unless they built their server with nand gates, I don't see running nginx reverse proxy to be incoherent with their philosophy.

Right, they only re-invent wheels when they feel they can make a better one. I don't think they feel they can go through the effort of fabricating a better chip. Though I wouldn't put it past rtm to try. The philosophy "Don't reinvent the wheel" however, is definitely inconsistent with their philosophy. They will reinvent the wheel whenever they feel they can make a better one. Just because they haven't reinvented ev…

I think he means that the nginx reverse proxy is just a part of the infrastructure, like the server, OS, MzScheme etc they use.

Re: Why HN was slow and how Rtm fixed it

#99

Post by mmaunder »

"In 7 seconds, a hundred or more connections accumulate. So the server ends up with hundreds of threads, most of them probably waiting for input (waiting for the HTTP request). MzScheme can be inefficient when there are 100s of threads waiting for input -- when it wants to find a thread to run, it asks the O/S kernel about each socket in turn to see if any input is ready, and that's a lot of asking per thread switch if there are lots of threads. So the server is able to complete fewer requests per second when there is a big backlog, which lets more backlog accumulate, and perhaps it takes a long time for the server to recover."

I may have misunderstood but it sounds like you have MzScheme facing the open internet? Try putting nginx (or another epoll/kqueue based server) in front of MzScheme. It will handle the thousands of connections you have that are waiting for IO with very little incremental CPU load and with a single thread. Then when nginx reverse proxies to MzScheme each request happens very fast because it's local which means you need much fewer threads for your app server. That means less memory and less of the other overhead that you get with a high thread count.

An additional advantage is that you can enable keepalive again (right now you have it disabled it looks like) which makes things a faster for first-time visitors. It also makes it slightly faster for us regulars because the conditional gets we do for the gif's and css won't have to reestablish connections. Less connections established means you give your OS a break too with fewer syn/syn-ack/ack TCP handshakes.

Someone mentioned below that reverse proxies won't work for HN. They mean that caching won't work - but a reverse proxy like nginx that doesn't cache but handles high concurrency efficiently should give you a huge perf improvement.

PS: I'd love to help implement this free. I run a 600 req/sec site using nginx reverse proxying to apache.

Re: Why HN was slow and how Rtm fixed it

#100

Post by mbubb »

I find it disturbing to see people asking "Who is Rtm?" "Who is filo?"

I understand if you are in tech you might not know figures in history or literature... but these guys?

Every time you login to a UNIX/Linux system you use the passwd file and related setup - authored at least in part by Rtm's father.

http://www.manpages.info/freebsd/passwd.1.html

Rtm has done lots in his own right as the wikipdia pages show.

But seriously - if you don't know who these people are you really should.

Read this: http://www.princeton.edu/~hos/Mahoney/unixhistory

and maybe ESR's writings and that online anthology of the early Apple days and old issues of 2600, etc, etc

I am sorry - but it is really irritating to me that someone would be on this site and really not be aware of the deeper history and culture. It is not that deep - 1950s to present (to cover Lisp).

As Jay-Z (whom you probably know) says - "Go read a book you illiterate son of a bitch and step up your vocab ..."

Post reply on HN