Live data from Hacker News

Serving 200M requests per day with a CGI-bin

jacob.gold

81–87 of 87 posts

Re: Serving 200M requests per day with a CGI-bin

#81

Surprised with the choice of Apache. There are better choices for serving CGI nowadays. The only reason for still running Apache is you have legacy cruft that requires Apache (like .htaccess).

Apache has a lot of cool modules for auth handling (like open connect), so this makes it easier if you want to write your business logic in a more obscure language while still have auth setup for you.

Re: Serving 200M requests per day with a CGI-bin

#82
post #79
post #77

Earlier quoted context omitted.

CGI is not anymore a major use case for Perl. The modern Perl web apps are built on PSGI and production deployment is as a long lived Perl process. I wouldn't be surprised to learn that Perl startup time has drifted. Need benchmark.

The major modern use case I know of is command-line utilities which also benefit from low start-up. Of course, that doesn't mean there hasn't been "perf rot" over the decades as you say. Such rot should never surprise anyone. :-) Some perl5 lover should take the time to compile all those 5.6 to 5.42 versions on the same host OS/CPU and do a performance comparison and create a nice chart for the world to trap and mayb…

Well, I spent a little more time on it, and for that one Perl version (on one cpu, i7-6700k, one OS - Linux 6.15, with one compiler gcc-15) I got a 1.5x slowdown of perl startup from 2008 to 2025. Not great!

    zsh# rei=(chrt 99 taskset -c 2-3 env -i HOME=$HOME PATH=/bin)
    zsh# tim=(-k2 -o14 -n14 -m14)
    
    # Set 1
    zsh# $rei tim $tim './perl-5.8.9
EDIT: Oh, yeah, and /n -> /dev/null is a symlink I always use.

Re: Serving 200M requests per day with a CGI-bin

#84
post #16
post #10

I got my start in the CGI era, and it baked into me an extremely strong bias against running short-lived subprocesses for things. We invented PHP and FastCGI mainly to get away from the performance hit of starting a new process just to handle a web request! It was only a few years ago that I realized that modern hardware means that it really isn't prohibitively expensive to do that any more - this benchmark gets to 2…

I think you might have found that CGI scripts deployed as statically-linked C binaries, with some attention given to size, you might've not been so disappointed. The "performance hit of starting a new process" is bigger if the process is a dynamically-linked php interpreter with gobs of shared libraries to load, and some source file, reading parsing compiling whatever, and not just by a little bit, always has been, s…

As soon as you need to open up a connection to a database, you are going to wish you had long-running processes again.

Re: Serving 200M requests per day with a CGI-bin

#86
Slow startup was definitely one reason to have long lived servers, but I’m surprised not to see the major other reasons:

- Keep-alive/pooled connections to remote services can significantly reduce average latency for making those calls.

- In-memory caches that allow amortizing repeated lookups across requests.

Just those two alone mean that a serious high performance server probably couldn’t get away with just CGI even ignoring startup time.

Re: Serving 200M requests per day with a CGI-bin

#87
post #71

Earlier quoted context omitted.

(I didn't downvote you) plow may not be the best tool that exists but it does make concurrent HTTP requests and generate metrics for them successfully. The writes returned 3xx because the handler returns a redirect, so this is expected.

> plow may not be the best tool that exists but it does make concurrent HTTP requests and generate metrics for them successfully. HTTP load testing is a problem area that is much more subtle than it seems. I've no doubt that plow does what you're saying here, but, without any tests whatsoever, I have serious doubts that it does so correctly, particularly if/when the load test starts bumping up against any of the nume…

You have a fair point about plow. I've used it enough to know that it basically works at spamming HTTP requests at a specified concurrency. When I'm doing something where I want better accuracy I've tended to use K6 and Vegeta.

> Yeah, but, unless `plow` actually follows that redirect, it's not really measuring the actual end-to-end latency...

Not in this case, since the purpose here was to measure the POST (write) not the subsequent GET (read) that a browser would do after the redirect.

Post reply on HN