11–20 of 20 posts
One thing this really shows the need for is a real api on heroku. Having to hardcode your username/pass into env variables is pretty ghetto. Hopefully heroku devs are watching and listening. Oauth plz??
Looks great. Hoping to see resque support on it soon. One thing this really shows the need for is a real api on heroku. Having to hardcode your username/pass into env variables is pretty ghetto. Hopefully heroku devs are watching and listening. Oauth plz??
Ha! And yeah, the ENV variables. :)
Nice work, glad to see heroku API gems getting some love. I wrote a similar gem back in January and have had good luck with it: https://github.com/freerobby/middle_management It's fully configurable via environment variables, so you can make changes without redeploying. It also lets you specify a minimum number of workers (even if there are zero jobs) so that you don't have to wait for workers to spin up if you don't…
Thanks for your input!
For the low-traffic scenario there's also an older, much simpler solution: https://github.com/mtravers/cheepnis
Nice work, glad to see heroku API gems getting some love. I wrote a similar gem back in January and have had good luck with it: https://github.com/freerobby/middle_management It's fully configurable via environment variables, so you can make changes without redeploying. It also lets you specify a minimum number of workers (even if there are zero jobs) so that you don't have to wait for workers to spin up if you don't…
Thanks. I like your approach of not having to deploy to change the min/max value of workers. Though it's hard to implement (I think) if you're working with a worker/ratio configuration like HireFire. Currently HireFire also supports the Mongoid ODM. I'm looking in to implementing the Resque worker as well so people can use HireFire with Resque with Redis on Heroku as well. Thanks for your input!
MiddleManagement works great for my needs, but if these libraries serve the same use case(s), I'd rather converge towards one library than have things spread out.
One other thing you may want to add is a retry loop for the heroku API calls. Every now and then, the heroku gem will throw a RestClient::ServiceUnavailable: 503 Service Unavailable exception. In the worst case, I'm not sure if this is best handled in the gem or at the application layer; however it's definitely worth retrying a couple times if you can avoid the blowup completely. I got some feedback from the Boston.rb group on handling this if you're interested: http://groups.google.com/group/boston-rubygroup/browse_threa...
The best I've found so far is: http://trevorturk.com/2011/01/21/delayed_job_admin-a-simple-...
Earlier quoted context omitted.
Couldn't agree more. The setup time of SimpleWorker is alot ( especially when dealing with specific gems ) and the fact that for DB work you have to create a separate non concurrent connection to the DB per job makes those times you talk about a reality. Ok, you could use the 'expensive jobs' but Heroku's DJ's win hands down in all tests I've done. Plus who doesn't like the handle_asynchronously :function.
Exactly. :) The DB connection is also quite frustrating depending on the location of the DB. MongoDB with MongoHQ or MongoMachine is fine since it's external already. But otherwise you have to know how to configure for example your MySQL PostgreSQL instances to allow incoming connections from external locations, and also becomes potentially insecure depending on your sysadmin skills. I find Heroku + DJ + HireFire so…
The isolated code issue has been a big issue, and we're addressing this through calls like merge_gem(), and soon merge_mailer(). It's a challenging task, and we see the gaps, but we're improving every day.
Runtime: this is also a top priority. We're working with each and every customer to ensure that their workers execute quickly. 3 seconds vs 30-60 is totally unacceptable and we won't leave you hanging here.
Queue speed: we're prototyping alternatives to http queuing which will dramatically speed up queue times.
This is just the tip of the iceberg for roadmap... Great work on HireFire. I see cases for both. Potentially massive parallelization on SimpleWorker (hundreds if not thousands of concurrent jobs), advanced scheduling needs, etc.
Anyways - we're listening - so any and all feedback is well received. If anybody has needs outside HireFire, in conjunction with, or otherwise, shoot me a line and we'll get you up and running at no cost to start.
Chad Arimura chad[at]simpleworker.com
Does anyone know of gems like this that will work on standard VPS servers? I'm launching something that completely depends on Delayed Job for processing data and would love an interface to manage queues and number of workers spun up at a given time. Also, visually watching the queue (a la resque) would be amazing. The best I've found so far is: http://trevorturk.com/2011/01/21/delayed_job_admin-a-simple-...
https://github.com/meskyanichi/hirefire/blob/develop/lib/hir...
Also, as far as I know that DJ Admin Interface is the only one for DJ that currently exists. If you want to use Resque, I'll be implementing Resque for HireFire soon.
Earlier quoted context omitted.
Exactly. :) The DB connection is also quite frustrating depending on the location of the DB. MongoDB with MongoHQ or MongoMachine is fine since it's external already. But otherwise you have to know how to configure for example your MySQL PostgreSQL instances to allow incoming connections from external locations, and also becomes potentially insecure depending on your sysadmin skills. I find Heroku + DJ + HireFire so…
Hey Guys, I'm a co-creator/founder of SimpleWorker.com. Can't say enough how appreciative we are of the feedback both good and bad. We've been experiencing pretty strong growth and through this are identifying the sticking points and working through them as fast as we can. The isolated code issue has been a big issue, and we're addressing this through calls like merge_gem(), and soon merge_mailer(). It's a challengin…
I think for me the hardest part was getting it to work the first time, waiting quite a while for the jobs to run to see if I managed to get it to work. The merge_gem is a huge improvement, though, handling credentials and re-connecting databases and other things are quite a hassle to set up (imo). I have to say that once it's all set up, and if the 30-60 start-up issue is gone, that it does seems like a beast for processing a lot of jobs concurrently and is exciting.
I'm definitely still looking forward to seeing future progress of SimpleWorker. So please do keep up the good work!
Earlier quoted context omitted.
Thanks. I like your approach of not having to deploy to change the min/max value of workers. Though it's hard to implement (I think) if you're working with a worker/ratio configuration like HireFire. Currently HireFire also supports the Mongoid ODM. I'm looking in to implementing the Resque worker as well so people can use HireFire with Resque with Redis on Heroku as well. Thanks for your input!
If I could fork your code and get it working with environment variables, would you be interested in pulling that in? (Not sure if you made a deliberate decision to avoid that). MiddleManagement works great for my needs, but if these libraries serve the same use case(s), I'd rather converge towards one library than have things spread out. One other thing you may want to add is a retry loop for the heroku API calls. Ev…
Thanks for your feedback!