31–40 of 202 posts
Sounds like there is a scalability issue within MzScheme in that it iterates over the number of threads, asking each thread about the sockets it has. As one can tell, once # of threads and # of sockets grow - finding which thread to run in user space becomes awfully expensive. As any clever admin will do, a least invasive fix involving limiting the number of connections and threads was done - with what sounds like im…
Who's Filo? David Filo?
Regardless of what you think of Yahoo's current situation, somebody who could easily retire wealthy but still hacks and flies economy class on Southwest to meetings in remote offices is worthy of respect.
> when [MzScheme] wants to find a thread to run, it asks the O/S kernel about each socket in turn to see if any input is ready They've never heard of select()? But really, is there some reason that it's hard to collect up all the fds at once or something?
Sounds like there is a scalability issue within MzScheme in that it iterates over the number of threads, asking each thread about the sockets it has. As one can tell, once # of threads and # of sockets grow - finding which thread to run in user space becomes awfully expensive. As any clever admin will do, a least invasive fix involving limiting the number of connections and threads was done - with what sounds like im…
not to mention that one thread per connection is, well, extremely outdated.
Earlier quoted context omitted.
flat files, no database
That's not the bottleneck. Essentially there's an in-memory database (known as hash tables). Stuff is lazily loaded off disk into memory, but most of the frequently needed stuff is loaded once at startup. The bottleneck is the amount of garbage created by generating pages. IIRC there is some horrible inefficiency involving UTF-8 characters.
> It turns out there is a hack in FreeBSD, invented by Filo, that causes the O/S not to give a new connection to the server until an entire HTTP request has arrived. This might reduce the number of threads a lot, and thus improve performance; I'll give it a try today or tomorrow. Anyone know if they're referring to "accept filters" here? FreeBSD folks can "man accf_http" if they're curious, which does prevent a reque…
Sounds like there is a scalability issue within MzScheme in that it iterates over the number of threads, asking each thread about the sockets it has. As one can tell, once # of threads and # of sockets grow - finding which thread to run in user space becomes awfully expensive. As any clever admin will do, a least invasive fix involving limiting the number of connections and threads was done - with what sounds like im…
http://www.cs.brown.edu/~sk/Publications/Papers/Published/kh...
I feel silly asking, but who or what is 'rtm'?