Live data from Hacker News

Announcing DJJob, a PHP port of delayed_job

seatgeek.com

11–15 of 15 posts

Re: Announcing DJJob, a PHP port of delayed_job

#11
post #9
post #6

In your example, does the call: $worker = new DJWorker($options); $worker->start(); Actually: 1. block until all jobs stored in the database have been completed 2. simply run a single job and return 3. return immediately and run all the jobs on a separate thread in the background I'm wondering where in a web application would be the best place to call $worker->start()?

I think the point is to run the worker outside of a web request and using a cron job

Or using the command line version of PHP and GNU Screen (I haven't fully studied the code, but it looks similar to a queue system that I hacked together a couple of days ago where the worker is wrapped in a while(TRUE) loop).

In the interest of some outside figures, how have you found this to scale? I implemented a similar solution as more of a stopgap until I had the time to get RabbitMQ installed with a "proper" worker dispatcher (more moving parts but less stress on the DB). I mostly ask because we implemented it for our notification system where we had to generate a send large batches of custom emails which was bogging down the page (and hard to debug where in the chain we were getting failed messages).

Re: Announcing DJJob, a PHP port of delayed_job

#12
post #9

Earlier quoted context omitted.

I think the point is to run the worker outside of a web request and using a cron job

Or using the command line version of PHP and GNU Screen (I haven't fully studied the code, but it looks similar to a queue system that I hacked together a couple of days ago where the worker is wrapped in a while(TRUE) loop). In the interest of some outside figures, how have you found this to scale? I implemented a similar solution as more of a stopgap until I had the time to get RabbitMQ installed with a "proper" wo…

Not sure who your question was directed to, but I've implemented a similar system in the past for a website with 10m+ users and 15+ web servers. It scaled just fine, we were sending close to 150K emails a day, among other background jobs for reporting, data cleanup, automated spam checks, etc

Re: Announcing DJJob, a PHP port of delayed_job

#13
post #10

I'm the author of Delayed Job. The popularity of it really surprises me to no end, and always brings a smile to my face. DJ took about 15 minutes to code up originally and is a product of one of those software rage moments ("why don't people queue like this??!"). I put it on GitHub because Jeremy Kemper really wanted to have a look at it. I thought the idea behind it was so naive that it would have occurred to everyo…

Great to hear from you. I remember being impressed by how short and straightforward delayed_job was when I first explored job queues. That's actually one of the only reasons I even considered building/porting something myself.

You're welcome, and thank you for delayed_job!

Re: Announcing DJJob, a PHP port of delayed_job

#14
post #5

you need to strip out the implementation specific classes such as propel and the mysql-specific calls to make this useful to anybody else for eg. an in-memory sqlite db would be ideal for this purpose (and a lot faster)

Absolutely, dropping Propel is TODO #1. I just figured I'd throw it on github and see if there was any interest first.

Re: Announcing DJJob, a PHP port of delayed_job

#15
Made something very similar to this in PHP about a year ago. It's buried in the Drupal project called Boost. It's the multi process crawler. It supports up to 8 processes crawling your site getting the cache warm. Gets call on cron & then 15 seconds after drupal's cron finished it starts up all 8 "threads". I've used it for image processing as well. Very stable will run for days if you give it a big enough queue (millions of pages to crawl). http://drupal.org/project/boost
Post reply on HN