Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is this how news.ycombinator.com runs? (Ctrl+F for "Closures Simulate Subroutines")

lib.store.yahoo.net

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Is this how news.ycombinator.com runs? (Ctrl+F for "Closures Simulate Subroutines")

#11

Post by papersmith »

malkia wrote: post #8
pg wrote: post #6

Earlier quoted context omitted.

For calls that don't need to know about more state than their arguments, we just use an ordinary url with arguments. Otherwise we make a closure and store it in a hash table in memory, and make the url be x?fnid= where the argument is the hash key. When an http request comes in, we call the corresponding closure. Sometimes what to do next after following a link generated by the closure (e.g. what to do after logging…

Isn't there a risk, that someone can (accidentally or not) enter hash-code of someone else's "closure"?

If access to the closures are session-specific, you'll have to hijack the session to do that.

Re: Is this how news.ycombinator.com runs? (Ctrl+F for "Closures Simulate Subroutines")

#12

Post by papersmith »

pg wrote: post #6
nostrademons wrote: post #2

I always assumed it was, and that the closures are stored in the user's session (in memory, on the server). You notice that if you leave a comment reply box up on the screen long enough, you'll get "Unknown or expired session" when you try to submit, and have to back up to the main comment page before you can try again?

For calls that don't need to know about more state than their arguments, we just use an ordinary url with arguments. Otherwise we make a closure and store it in a hash table in memory, and make the url be x?fnid= where the argument is the hash key. When an http request comes in, we call the corresponding closure. Sometimes what to do next after following a link generated by the closure (e.g. what to do after logging…

"in that case it's a bit like a continuation."

Doesn't MzScheme (assuming it's what Arc is built on) have first-class continuation already?

Re: Is this how news.ycombinator.com runs? (Ctrl+F for "Closures Simulate Subroutines")

#13

Post by pg »

ced wrote: post #4
nostrademons wrote: post #2

I always assumed it was, and that the closures are stored in the user's session (in memory, on the server). You notice that if you leave a comment reply box up on the screen long enough, you'll get "Unknown or expired session" when you try to submit, and have to back up to the main comment page before you can try again?

[silly test to see if they remove the closure from the HT once it's used, and the answer seems no]

Correct, unless the closure explicitly removes itself when executed, of course.

Re: Is this how news.ycombinator.com runs? (Ctrl+F for "Closures Simulate Subroutines")

#14

Post by pg »

papersmith wrote: post #12
pg wrote: post #6

Earlier quoted context omitted.

For calls that don't need to know about more state than their arguments, we just use an ordinary url with arguments. Otherwise we make a closure and store it in a hash table in memory, and make the url be x?fnid= where the argument is the hash key. When an http request comes in, we call the corresponding closure. Sometimes what to do next after following a link generated by the closure (e.g. what to do after logging…

"in that case it's a bit like a continuation." Doesn't MzScheme (assuming it's what Arc is built on) have first-class continuation already?

It does, and so does Arc, but I don't think we've needed them in anything we've written so far. News.YC is a pretty simple application.

Re: Is this how news.ycombinator.com runs? (Ctrl+F for "Closures Simulate Subroutines")

#15

Post by shiro »

papersmith wrote: post #12
pg wrote: post #6

Earlier quoted context omitted.

For calls that don't need to know about more state than their arguments, we just use an ordinary url with arguments. Otherwise we make a closure and store it in a hash table in memory, and make the url be x?fnid= where the argument is the hash key. When an http request comes in, we call the corresponding closure. Sometimes what to do next after following a link generated by the closure (e.g. what to do after logging…

"in that case it's a bit like a continuation." Doesn't MzScheme (assuming it's what Arc is built on) have first-class continuation already?

From my experience of continuation-based web app, what you're dealing with is not a full continuation captured by call/cc, but rather a delimited (partial) continuation, which captures the continuation of your application logic but not the state of underlying protocols. You can implement delimited continuations on top of call/cc, and I did based on Gasbichler&Sperber paper (ICFP02), but in most cases explicit CPS wrapped in some macros are just as well worked.

Re: Is this how news.ycombinator.com runs? (Ctrl+F for "Closures Simulate Subroutines")

#16

Post by damir »

pg wrote: post #14
papersmith wrote: post #12

Earlier quoted context omitted.

"in that case it's a bit like a continuation." Doesn't MzScheme (assuming it's what Arc is built on) have first-class continuation already?

It does, and so does Arc, but I don't think we've needed them in anything we've written so far. News.YC is a pretty simple application.

Can we expect to see Arc go public in the near future?

Re: Is this how news.ycombinator.com runs? (Ctrl+F for "Closures Simulate Subroutines")

#17

Post by nickb »

Paul, could you enlighten us how well does this type of development (Lisp macro/closure based) integrate with JS and Ajax? I'm interested in asynchronicity of the Ajax... Ruby and Python with Protoype and Scriptaculous make this very easy indeed...

Re: Is this how news.ycombinator.com runs? (Ctrl+F for "Closures Simulate Subroutines")

#18

Post by pg »

nickb wrote: post #17

Paul, could you enlighten us how well does this type of development (Lisp macro/closure based) integrate with JS and Ajax? I'm interested in asynchronicity of the Ajax... Ruby and Python with Protoype and Scriptaculous make this very easy indeed...

How much Ajax do you see on this site? (The answer is that I don't know, because I haven't tried.)

Re: Is this how news.ycombinator.com runs? (Ctrl+F for "Closures Simulate Subroutines")

#19

Post by ralph »

pg wrote: post #6
nostrademons wrote: post #2

I always assumed it was, and that the closures are stored in the user's session (in memory, on the server). You notice that if you leave a comment reply box up on the screen long enough, you'll get "Unknown or expired session" when you try to submit, and have to back up to the main comment page before you can try again?

For calls that don't need to know about more state than their arguments, we just use an ordinary url with arguments. Otherwise we make a closure and store it in a hash table in memory, and make the url be x?fnid= where the argument is the hash key. When an http request comes in, we call the corresponding closure. Sometimes what to do next after following a link generated by the closure (e.g. what to do after logging…

What extra data is in the closure for "reply" that means it isn't a simple URL? Is it just what to do after the reply is complete, i.e. what post to return to viewing? Is there a reason this isn't taken from the Referer instead?

I guess this technique doesn't scale too well unless the hash table is either shared across servers or a user is kept on one server, which then causes problems of its own.

I'm guessing 20,000 closures is plenty and I've been bitten by server restarts. Out of curiosity, do you keep track of the minimum age of the 20,000th hash entry prior to deleting to make room for another? If that's something huge, it would confirm server restarts are the normal reason the "unknown fnid" message is seen.

Re: Is this how news.ycombinator.com runs? (Ctrl+F for "Closures Simulate Subroutines")

#20

Post by ralph »

pg wrote: post #6
nostrademons wrote: post #2

I always assumed it was, and that the closures are stored in the user's session (in memory, on the server). You notice that if you leave a comment reply box up on the screen long enough, you'll get "Unknown or expired session" when you try to submit, and have to back up to the main comment page before you can try again?

For calls that don't need to know about more state than their arguments, we just use an ordinary url with arguments. Otherwise we make a closure and store it in a hash table in memory, and make the url be x?fnid= where the argument is the hash key. When an http request comes in, we call the corresponding closure. Sometimes what to do next after following a link generated by the closure (e.g. what to do after logging…

Having thought about this a little more, doesn't this approach lay you open to an easy denial of service attack? I may not have the resources of CPU or bandwidth to hammer a site using this technique, but it seems that the fnids are unique to the page they're generated for, perhaps because the time is part of the key, so by requesting a page enough times I can soon use up the 20,000 closures. This would cause "unknown fnid" errors for other users despite them having only recently generated the page.

Here's why I think this. Apologies for the formatting.

$ fnid() { wget -qO - "http://news.ycombinator.com/threads?id=${1?}" | tr '?"' '\n\n' | sed -n 's/^fnid=//p'; }

$ fnid ralph | sort 1

$ fnid ralph | sort 2

$ wc -l 1 2 | tr \\n ,

177 1, 177 2, 354 total,

$ comm 1 2 | sed 's/[^\t].//; s/./x/g; s/^/x/' | sort | uniq -c | tr \\n ,

177 x, 177 xx,

comm(1) is showing the two fnid lists are disjoint. 177 occur only in file 1. 177 occur only in file 2. 0 occur in both (xxx). 20,000 / 177 = 112.99, so 113 "fnid ralph" would flush the hash table.

I'm not trying to pick holes, just interested in the techniques that can be used and would welcome opinions.

Post reply on HN