101–110 of 202 posts
I find it disturbing to see people asking "Who is Rtm?" "Who is filo?" I understand if you are in tech you might not know figures in history or literature... but these guys? Every time you login to a UNIX/Linux system you use the passwd file and related setup - authored at least in part by Rtm's father. http://www.manpages.info/freebsd/passwd.1.html Rtm has done lots in his own right as the wikipdia pages show. But s…
I feel silly asking, but who or what is 'rtm'?
pg's friend. YC cofounder. http://en.wikipedia.org/wiki/Robert_Tappan_Morris
The traffic graphs linked in this post [0] are an interesting addition to the "How often do you visit HN?" poll [1] that was done a week ago. From the graphs, it looks like there are about 10x as many page views as unique IPs. [0] http://ycombinator.com/images/hntraffic-17jan11.png [1] http://news.ycombinator.com/item?id=2090191
Earlier quoted context omitted.
Read C10K? Both select() and poll() have this problem internally. You have to use one of the more advanced techniques available if you really want to scale. epoll(), kqueue() or friends.
Rtm's phrasing implies that MzScheme is making a syscall for each socket in turn, so it sounds like it isn't even using the basic select() or poll().
Since no one has mentioned it yet - Varnish-cache.org, written by a FreeBSD kernel hacker, has a very nice feature, in that it will put all overlapping concurrent requests for the same cacheable resource "on hold", only fetch that resource once from the backend, then serve the same copy to all. Nearly all the expensive content on HN would be cacheable by varnish. Then you can get it down to pretty close to "1 backend…
Reverse proxies won't work for HN, because requests for the same resource from multiple users can't use the same results. Not only are certain bits of info customized for the user (like your name/link at the top), but even things like the comments and links are custom per user. Things like users' showdead value, as well as whether the user is deaded, can drastically change the output of each page. Eg, comments by a d…
sub vcl_hash { set req.hash += req.http.cookie; }
What this means is that the cache is per-logged-in-user and pretty much personalized. The server's going to need a lot more RAM than usual. You can set a low TTL on the cache entries so they're flushed and not kept in memory indefinitely. But the performance boost is great.
This is not recommended as an always-on measure. We wrote an entry about accomplishing something similar w/ python&varnish. Here it is if you're interesting in reading about it: http://blog.unixy.net/2010/11/3-state-throttle-web-server/
Regards
Earlier quoted context omitted.
Reverse proxies won't work for HN, because requests for the same resource from multiple users can't use the same results. Not only are certain bits of info customized for the user (like your name/link at the top), but even things like the comments and links are custom per user. Things like users' showdead value, as well as whether the user is deaded, can drastically change the output of each page. Eg, comments by a d…
Varnish supports edge side includes. The header bar could be an ESI and the rest of the page could be cached
Except they can't, for the reasons I mentioned above. Eg, if my account is deaded, when I view a thread with one of my own comments, it looks different than if someone else was viewing that some thread, especially for those of us with or without showdead checked in our profiles.
Its not as straightforward as you would like it to be.
> In 7 seconds, a hundred or more connections accumulate. So
> the server ends up with hundreds of threads, most of them
> probably waiting for input
Earlier quoted context omitted.
Feel free to fork MzScheme and replace the garbage collector with a new one that runs continuously.
Surely there's a way to target the JVM? I'd trust my life on the JVM, it's pretty battle tested, and the GC is simply awesome.
You sure you want to do that? :-)
> Java technology is not fault tolerant and is not designed, manufactured, or intended for use or resale as on-line control equipment in hazardous environments requiring fail-safe performance, such as in the operation of nuclear facilities, aircraft navigation or communication systems, air traffic control, direct life support machines, or weapons systems, in which the failure of Java technology could lead directly to death, personal injury, or severe physical or environmental damage
Earlier quoted context omitted.
Can't you only use varnish for mostly non-dynamic content? Like for example, wouldn't the fact that it displays my username and karma score at the top of the page make it so that you couldn't use varnish (or at least make it more difficult)?
Yes, you'd have to somehow separate the dynamic content from the static content so they could be fetched in different requests and then combined (probably via ajax). If it's just your username and karma then it's simple enough, but if the comments are displayed differently for different people then it could be bear.
Aside from the header, there's only a relatively small number of variations for any given content, right? Showdead, ability to downvote, etc? So, each of these variations gets a distinct ESI URL. Like /item_$showdead_$downvote_$etc right in the internal URL, so any combination of these is a distinct URL. Only the first user to hit any particular combination would result in a request to the backend, and that could remain in cache until its content changed. No wizardry required.
Since no one has mentioned it yet - Varnish-cache.org, written by a FreeBSD kernel hacker, has a very nice feature, in that it will put all overlapping concurrent requests for the same cacheable resource "on hold", only fetch that resource once from the backend, then serve the same copy to all. Nearly all the expensive content on HN would be cacheable by varnish. Then you can get it down to pretty close to "1 backend…
You clearly don't understand the problem. Even mod_pagespeed or memcached would be more appropriate here: They are rate-limited by the LISP kernel anyway (we are talking about dynamic content here).