I think the article is good, however, I feel compelled to weigh in from the countervailing general direction: I am a message queue skeptic. Not that they should never be used, but rather a general feeling that complex, dedicated message queue software is often used for engineering problems between two or three orders of magnitudes too small before they deliver value. And, for most projects, queue replacement is not t…
Several years ago (probably first or second year of high school) I wrote and distributed amongst my friends a small desktop game. The program e-mailed me their new high-scores and internal game diagnostics daily. I found that this was too often so I rewrote the game to send these data daily. This way of retrieving data sucked. Often, I would get empty or unimportant messages.
Eventually I had the realisation that time-oriented polling was the wrong philosophy. Instead, there should be a "this event has happened now" (e.g. 500kb of diag. generated) algorithm that calls the polling routine/invokes the 'stuff to be done traverser'. If I understand the problem correctly, the mass e-mail example, you could make a simple counter as part of the HTTP request. When this counter hits a certain you-determined threshold value (e.g. there are 500 jobs to do), then call the traverse/processing code:
if jobs_to_do > thresh then invoke some async processing
The takeaway from this post is that, in my opinion, time-based (cron job-bish) polling algorithms are inefficient but can be replaced with a rudimentary event-driven ones.
And just BTW, 'countervailing' - I'm deifying this word. Holy s*. I'm going to use it all the time.