Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asynchronous Processing in Web Apps, Part 1: A Database Is Not a Queue

blog.gomiso.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Asynchronous Processing in Web Apps, Part 1: A Database Is Not a Queue

#21

Post by joevandyk »

icebraining wrote: post #9

With a traditional database this typically means a service that is constantly querying for new processing tasks or messages. Traditionally, but not necessarily; PostgreSQL supports the LISTEN and NOTIFY commands for asynchronous notifications, without polling.

queue_classic is a postgresql job queue that can use listen/notify. http://www.postgresql.org/docs/9.1/static/sql-notify.html

Although queue_classic doesn't use listen/notify by default, and I'm not sure why. I think it should, it seems to perform better.

Re: Asynchronous Processing in Web Apps, Part 1: A Database Is Not a Queue

#22

Post by desas »

blcArmadillo wrote: post #16

Look forward to reading the future articles. Right now I'm working on a side project that has a need for some asynchronous tasks. I was planning on using beanstalkd but the one thing that concerns me is that if the queue goes down the outstanding jobs are not persisted. Any recommendations on the best way around this?

beanstalkd -b should work. I haven't tried this.

Re: Asynchronous Processing in Web Apps, Part 1: A Database Is Not a Queue

#23

Post by nesquena »

icebraining wrote: post #9

With a traditional database this typically means a service that is constantly querying for new processing tasks or messages. Traditionally, but not necessarily; PostgreSQL supports the LISTEN and NOTIFY commands for asynchronous notifications, without polling.

I agree and if polling was the only consideration then this would mitigate the issue. Admittedly I didn't cover this as much in my article but there's also a lot of other flexibility afforded by a good message queue around delivery strategies, consumption strategies, error handling, etc that you would have to do much more manually and painfully trying to shoehorn it into a database (even a good one). I am not saying PostgreSQL notify commands are not useful for certain tasks, just encouraging people to understand the tools available and pick the right one for the job.

Re: Asynchronous Processing in Web Apps, Part 1: A Database Is Not a Queue

#24

Post by nesquena »

blcArmadillo wrote: post #16

Look forward to reading the future articles. Right now I'm working on a side project that has a need for some asynchronous tasks. I was planning on using beanstalkd but the one thing that concerns me is that if the queue goes down the outstanding jobs are not persisted. Any recommendations on the best way around this?

Yes beanstalkd has solid persistence support now in later versions. You can use the “-b” option, and beanstalkd will write all jobs to a binlog. If the power goes out, you can restart beanstalkd with the same option and it will recover the contents of the log.

Re: Asynchronous Processing in Web Apps, Part 1: A Database Is Not a Queue

#25

Post by nesquena »

matticakes wrote: post #14

disclaimer: co-author of NSQ [1] here Agreed. Message queues play an important role for us (bitly) in being a layer of fault tolerance, buffering, and a means to perform various operational tasks. They're so important to us that we decided to build something that worked exactly like we wanted. NSQ is a realtime distributed message processing system where we've taken the approach of focussing on making it ops friendly…

Interesting, thanks for sharing the links. I think building your own message queue is sometimes the only way to get things working exactly as you might want. For people looking for a highly configurable framework for building a tailored MQ, be sure to check out http://www.zeromq.org/ as a basis.

Re: Asynchronous Processing in Web Apps, Part 1: A Database Is Not a Queue

#26

Post by nesquena »

jhartmann wrote: post #7

Nice article. One thing you should talk about in my opinion are systems like Redis. Redis can be used as a generalized key value store, but it can also be used as a messaging platform. In fact many systems like Storm (which would be another great topic) have easy integration with redis pub subs. While Redis and other solutions like it probably are not a good fit for all your data, they are great for mixed supporting…

Definitely, thanks for your feedback. Will be bringing up redis in the next post. Redis has some messaging functionality baked right in and can be a good solution as seen with https://github.com/defunkt/resque in Ruby as well. ZeroMQ is also an important technology to understand in the context of this discussion.

Re: Asynchronous Processing in Web Apps, Part 1: A Database Is Not a Queue

#27

Post by seunosewa »

lsh123 wrote: post #19
rhettg wrote: post #18

Earlier quoted context omitted.

I understood the point as "Don't use a database as a replacement for a queue". Use the right tool for the job. The fact that gearmand is backed by a DB is not at all the same as using a DB for a queue directly. Gearmand just uses a database as a backup that it can reload tasks when it get's restarted.

Yes. However, you have same issues with DB tables receiving tons of inserts/deletes/selects. This is the worst possible DB load :)

DB's are designed for this purpose.

Re: Asynchronous Processing in Web Apps, Part 1: A Database Is Not a Queue

#28

Post by randomtree »

If you would consider using NoSQL, then MongoDB might be a good fit. It has a messaging queue. It's called capped collections with tailable cursors (http://www.mongodb.org/display/DOCS/Tailable+Cursors ). It's persistent, you don't need polling, and you don't need to remove processed messages.

Re: Asynchronous Processing in Web Apps, Part 1: A Database Is Not a Queue

#29

Post by cico71 »

chris_wot wrote: post #13
fdr wrote: post #5

I think the article is good, however, I feel compelled to weigh in from the countervailing general direction: I am a message queue skeptic. Not that they should never be used, but rather a general feeling that complex, dedicated message queue software is often used for engineering problems between two or three orders of magnitudes too small before they deliver value. And, for most projects, queue replacement is not t…

Interestingly, that's why Microsoft created SQL Server Service Broker. They had te infrastructure for reliable message queuing and transactional support, so they created SQL Server Service Broker! Not sure I ever took off though...

Unfortunately it didn't really took off, but it's a shame because it's a good, polished and complete implementation that allows for some advanced scale-out topologies (it can also be used as a foundation for data dependent routing and map/reduce scenarios).

I guess it's not much used because people like to reinvent the wheel every time (by manually implementing queues using tables with all the traditional concurrency problems) instead of learning something a bit more complex.

Anyway, it's not going to be thrown away anytime soon as it's used in other parts of the engine (e.g. SMTP mail integration, Server Events and Query Notifications).

Post reply on HN