Disque 1.0 RC1 is out • Hacker News Board

Re: Disque 1.0 RC1 is out

#51

Post by manigandham » Sun, Jan 03, 2016, 4:52 PM UTC

nemothekid wrote: post #45

lobster_johnson wrote: post #17

Earlier quoted context omitted.

I would argue that job management is unrelated to messaging, at least according my loose definition of a task that: * Has a well-defined lifetime — unstarted, running, paused, completed successfully, or failed; * Is executed from some kind of parameterized "job specification" that describes its inputs and desired behavior; * Has state data (e.g., completion % progress, log output, metrics, transactional continutation…

My gripe with using a relational database as a priority queue is tends not to scale well. (Similarly as if you tried to use a relational database as a queue). Once you get to the 10s of millions of messages per day (for example ~8M at priority X and 2M at priority Y), performance tends to go down the drain (due to contention), disk space bloats (due to vacumming/deletes). I've yet to come across a good task schedulin…

Parent is only talking about keeping the job state in a database, not the queue messages.

Re: Disque 1.0 RC1 is out

#52

Post by ksec » Sun, Jan 03, 2016, 5:30 PM UTC

Idle observation, The blog shows what simple, text, and minimal graphics / pics ( In this case there are Zero ) can load incredibly fast...

Re: Disque 1.0 RC1 is out

#53

Post by kfunk » Sun, Jan 03, 2016, 6:55 PM UTC

vdfs wrote: post #8

Is there a product-ready alternative to Disque?

We have used nsq to ship billions of log messages daily for over 2 years. It is very sensible operationly and easy to configure to the level of robustness your use may have. It does not allow seeking into the queue like Kafka, but we have never needed that capability, and we found it easier to design around this rather than take on the operational burden.

Re: Disque 1.0 RC1 is out

#54

Post by nemothekid » Sun, Jan 03, 2016, 8:02 PM UTC

manigandham wrote: post #51

nemothekid wrote: post #45

Earlier quoted context omitted.

My gripe with using a relational database as a priority queue is tends not to scale well. (Similarly as if you tried to use a relational database as a queue). Once you get to the 10s of millions of messages per day (for example ~8M at priority X and 2M at priority Y), performance tends to go down the drain (due to contention), disk space bloats (due to vacumming/deletes). I've yet to come across a good task schedulin…

Parent is only talking about keeping the job state in a database, not the queue messages.

How does that work? If the the queue is ultimately delivering the messages, doesn't that mean you can't have priority based queuing?