Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why HN was slow and how Rtm fixed it

ycombinator.com

Post reply on HN
Print view View original

191–200 of 202 posts

Re: Why HN was slow and how Rtm fixed it

#191

Post by [deleted] »

gchpaco wrote: post #153
sedachv wrote: post #115

Earlier quoted context omitted.

Or I don't know, use continuations in a place that's actually appropriate? John Fremlin showed that even with horrible CPS rewriting and epoll you can get way better throughput in SBCL (TPD2) than nginx. MzScheme comes with native continuations. It's not hard to call out to epoll. Instead everyone in the Lisp community (pg included) is still enamored with using continuations to produce ugly URLs and unmaintainable we…

MzScheme/Racket's continuations are of the "copy the C stack" variety, or were last time I checked. They are in no way efficient; it would probably be better to CPS transform your own code than try to use MzScheme/Racket's continuations directly in performance sensitive code.

[deleted]

Re: Why HN was slow and how Rtm fixed it

#192

Post by marcamillion »

axod wrote: post #186
jmtulloss wrote: post #184

Earlier quoted context omitted.

Neither Python nor PHP run garbage collection continuously in a separate thread. Python is reference counted, and as far as I know PHP is too.

But I'd bet both of them perform far better than MzScheme. In any event, it's a "solved problem". The sheer amount of time and effort going into this is silly. Just pick a better language/platform and use it.

I guess the point is, axod, if you aren't willing to backup talk with action, it might be best to think about what you say before you say it.

Don't slam something saying I could do better, and then have your bluff called.

Makes you look silly. This applies to even if the person calling your bluff was NOT pg.

Re: Why HN was slow and how Rtm fixed it

#193

Post by pg »

jules wrote: post #188
pg wrote: post #176

Earlier quoted context omitted.

I've been using this technique since 1995 and it has never once been a problem. It's an instance of programming with closures, which has been common in the Lisp world for even longer. One doesn't need to examine something represented as a closure any more than one needs to examine a particular invocation of a recursive function. Perhaps the reason I've never had a problem is that I've been careful to use this techniq…

> I've been using this technique since 1995 and it has never once been a problem. Thousands of HN users experience the problems every day: link expired.

The issue we were talking about was maintainability.

Giving links longer expirations is trivially easy, and I already do it in cases where it matters, like submit buttons on big forms.

Re: Why HN was slow and how Rtm fixed it

#194

Post by pg »

blasdel wrote: post #182
pg wrote: post #176

Earlier quoted context omitted.

I've been using this technique since 1995 and it has never once been a problem. It's an instance of programming with closures, which has been common in the Lisp world for even longer. One doesn't need to examine something represented as a closure any more than one needs to examine a particular invocation of a recursive function. Perhaps the reason I've never had a problem is that I've been careful to use this techniq…

It's a problem for me when the fnids in every reply cause them to expire several times a day when the server crashes. Edit: also when the server redirects me back to the wrong origin, I was sent to http://news.ycombinator.com/threads?id=pg instead of http://news.ycombinator.com/item?id=2120756 after posting this reply initially.

also when the server redirects me back to the wrong origin

Sounds like that could be a bug. Was http://news.ycombinator.com/item?id=2120756 the page you were on when you replied?

Re: Why HN was slow and how Rtm fixed it

#195

Post by jules »

pg wrote: post #193
jules wrote: post #188

Earlier quoted context omitted.

> I've been using this technique since 1995 and it has never once been a problem. Thousands of HN users experience the problems every day: link expired.

The issue we were talking about was maintainability. Giving links longer expirations is trivially easy, and I already do it in cases where it matters, like submit buttons on big forms.

The problem is that links expire at all. There are two products: HN with "link expired" and HN without "link expired". You can write the former in a highly maintainable way.

Also, even if you accept that links expire, it's not trivial to make the links not expire for a long time. You can add a lot of RAM to a single machine, but only up to a point. Supporting multiple machines is very hard, although it can be done (distributed object system). RAM is not the only problem however. The server is inevitably going to restart once in a while. Perhaps not in the literal sense of killing and restarting the mzscheme process if you're very careful, but still in the practical sense. The data structures for storing content change as you develop a web app/site, thereby invalidating old closures hanging around.

Re: Why HN was slow and how Rtm fixed it

#196

Post by blasdel »

pg wrote: post #194
blasdel wrote: post #182

Earlier quoted context omitted.

It's a problem for me when the fnids in every reply cause them to expire several times a day when the server crashes. Edit: also when the server redirects me back to the wrong origin, I was sent to http://news.ycombinator.com/threads?id=pg instead of http://news.ycombinator.com/item?id=2120756 after posting this reply initially.

also when the server redirects me back to the wrong origin Sounds like that could be a bug. Was http://news.ycombinator.com/item?id=2120756 the page you were on when you replied?

Yes, I clicked the reply link on your comment from there.

I figured you had to know about this bug, since it happens to me regularly (maybe 10% of eligible comments) when I comment in active threads during standard procrastination hours. The misdirect is usually to the threads page of a user further up the comment tree, though sometimes it's to the permalink of a grandparent comment.

Seems like you're mixing up the redirects of concurrent users but never across comment hierarchies so it's not omnipresent.

Re: Why HN was slow and how Rtm fixed it

#197

Post by akkartik »

mmaunder wrote: post #99

"In 7 seconds, a hundred or more connections accumulate. So the server ends up with hundreds of threads, most of them probably waiting for input (waiting for the HTTP request). MzScheme can be inefficient when there are 100s of threads waiting for input -- when it wants to find a thread to run, it asks the O/S kernel about each socket in turn to see if any input is ready, and that's a lot of asking per thread switch…

"it sounds like you have MzScheme facing the open internet?"

Yep. The arc webserver runs directly on port 80, for which it needs to run as root. To avoid all sorts of security headaches, it runs:

  (setuid 2)
soon after startup.

The whole thing seems hacky.

Re: Why HN was slow and how Rtm fixed it

#198

Post by jmtulloss »

axod wrote: post #186
jmtulloss wrote: post #184

Earlier quoted context omitted.

Neither Python nor PHP run garbage collection continuously in a separate thread. Python is reference counted, and as far as I know PHP is too.

But I'd bet both of them perform far better than MzScheme. In any event, it's a "solved problem". The sheer amount of time and effort going into this is silly. Just pick a better language/platform and use it.

It depends on what you mean by "perform better". Any reference-counted GC mechanism will have memory leaks, which is a serious problem in a long-running process. Python gets around this by also occasionally running a traditional GC, which, btw, is a stop-the-world GC.

Re: Why HN was slow and how Rtm fixed it

#199

Post by sayrer »

mmaunder wrote: post #99

"In 7 seconds, a hundred or more connections accumulate. So the server ends up with hundreds of threads, most of them probably waiting for input (waiting for the HTTP request). MzScheme can be inefficient when there are 100s of threads waiting for input -- when it wants to find a thread to run, it asks the O/S kernel about each socket in turn to see if any input is ready, and that's a lot of asking per thread switch…

They said caching wouldn't work, but they could be wrong. You can't change the Cache-Control header to public for HN responses, because the same URL can appear different to different users. There may be some ways around this, including giving each logged-in user their own URL to browse with.

But that might be a lot of work. You can still set up a proxy that kicks in only for requests that don't contain a session cookie. Then, requests without a cookie can be responded to with a cached copy from Varnish, and Varnish could refresh every 30 seconds or so. That might reduce the number of connections to MzScheme by quite a lot.

Re: Why HN was slow and how Rtm fixed it

#200

Post by projectileboy »

defen wrote: post #108
axod wrote: post #93

Earlier quoted context omitted.

Surely there's a way to target the JVM? I'd trust my life on the JVM, it's pretty battle tested, and the GC is simply awesome.

> I'd trust my life on the JVM You sure you want to do that? :-) > Java technology is not fault tolerant and is not designed, manufactured, or intended for use or resale as on-line control equipment in hazardous environments requiring fail-safe performance, such as in the operation of nuclear facilities, aircraft navigation or communication systems, air traffic control, direct life support machines, or weapons system…

I've helped build rail control systems that run on Java, so you're probably trusting your life to Java daily without knowing it.
Post reply on HN