Disque 1.0 RC1 is out • Hacker News Board

Re: Disque 1.0 RC1 is out

#21

Post by boomzilla » Sat, Jan 02, 2016, 8:52 PM UTC

Jasper_ wrote: post #12

lobster_johnson wrote: post #6

Disque is definitely exciting, and looks like it can replace RabbitMQ, which has serious flaws in its clustering design. I'm looking forward to trying it out. However, if some constructive criticism is permitted, I have to say that, having written distributed applications for many years, I have come to dislike the "classical" push/pop queue data model: * Acking is a bad idea. It requires the broker to manage a lot of…

Kafka has a different data model which works for some scenarios, but not others. We attempted to use Kafka as part of a job management system, where long-running jobs were scheduled and workers consumed partitions, but what we found was that since consumers work on partitions, a long-running task could block an entire partition's worth of work, with no way to migrate it to another partition. Kafka works really well w…

What are the downsides with SQS if you expect a small number of large massages (well, not larger than the 4K limit I think)? The bill should be a lot cheaper than running a broker your self. Are you worried about lock-in? SQS APIs are simple enough to replicate with a REST service backed by a database for low usage scenarios.

Re: Disque 1.0 RC1 is out

#22

Post by tbrock » Sat, Jan 02, 2016, 9:02 PM UTC

AznHisoka wrote: post #14

vdfs wrote: post #8

Is there a product-ready alternative to Disque?

I've been using Sidekiq for the past 2 years, and deem it as production ready. It does force you to use Redis and Ruby though.

I'm building a node.js application now and WISH we had something half as good as Sidekiq written in Javascript.

Re: Disque 1.0 RC1 is out

#23

Post by lobster_johnson » Sat, Jan 02, 2016, 9:05 PM UTC

Jasper_ wrote: post #20

lobster_johnson wrote: post #17

Earlier quoted context omitted.

I would argue that job management is unrelated to messaging, at least according my loose definition of a task that: * Has a well-defined lifetime — unstarted, running, paused, completed successfully, or failed; * Is executed from some kind of parameterized "job specification" that describes its inputs and desired behavior; * Has state data (e.g., completion % progress, log output, metrics, transactional continutation…

Sure, that's actually very similar to the the system we built internally. But I don't see how Kafka as a messaging layer helps in your system, either. Presumably, the consumers won't advance their offset in their partition until the job is done, because they are not ready to start a new job. So you will still get the same blocking behavior. In addition, I don't see how you implement priority in a system like this, si…

I didn't mean to imply that Kafka was suitable for such a system, though that's mostly because it doesn't support priority queues. You could work around this by using a fixed range of levels and one queue per priority level, though it would probably just complicate things.

Re: Disque 1.0 RC1 is out

#24

Post by tetron » Sat, Jan 02, 2016, 9:24 PM UTC

lobster_johnson wrote: post #17

Jasper_ wrote: post #12

Earlier quoted context omitted.

Kafka has a different data model which works for some scenarios, but not others. We attempted to use Kafka as part of a job management system, where long-running jobs were scheduled and workers consumed partitions, but what we found was that since consumers work on partitions, a long-running task could block an entire partition's worth of work, with no way to migrate it to another partition. Kafka works really well w…

I would argue that job management is unrelated to messaging, at least according my loose definition of a task that: * Has a well-defined lifetime — unstarted, running, paused, completed successfully, or failed; * Is executed from some kind of parameterized "job specification" that describes its inputs and desired behavior; * Has state data (e.g., completion % progress, log output, metrics, transactional continutation…

This is basically the architecture of Arvados Crunch: http://arvados.org

Re: Disque 1.0 RC1 is out

#25

Post by kevan » Sat, Jan 02, 2016, 9:28 PM UTC

> However I’m not living into the illusion that I got everything right in the first release, so it will take months (or years?) of iteration to really reach the operational simplicity I’m targeting.

It's always refreshing to hear such good programmers acknowledging how hard building complex systems is.

Re: Disque 1.0 RC1 is out

#26

Post by BogusIKnow » Sat, Jan 02, 2016, 9:45 PM UTC

kevan wrote: post #25

> However I’m not living into the illusion that I got everything right in the first release, so it will take months (or years?) of iteration to really reach the operational simplicity I’m targeting. It's always refreshing to hear such good programmers acknowledging how hard building complex systems is.

And this from antirez who writes such excellent programms, Redis runs in several companies I've been, without ever crashing, making any trouble - most people I've met forget they have Redis running because it just works and works and works. Can't praise that piece of code high enough.

Re: Disque 1.0 RC1 is out

#27

Post by true_religion » Sat, Jan 02, 2016, 9:50 PM UTC

tbrock wrote: post #22

AznHisoka wrote: post #14

Earlier quoted context omitted.

I've been using Sidekiq for the past 2 years, and deem it as production ready. It does force you to use Redis and Ruby though.

I'm building a node.js application now and WISH we had something half as good as Sidekiq written in Javascript.

How would you say it compares to something like Celery?

Re: Disque 1.0 RC1 is out

#28

Post by bipin_nag » Sat, Jan 02, 2016, 10:12 PM UTC

Looking forward to Aphyr's Call Me Maybe for Disque.

Re: Disque 1.0 RC1 is out

#29

Post by tbrock » Sat, Jan 02, 2016, 10:16 PM UTC

true_religion wrote: post #27

tbrock wrote: post #22

Earlier quoted context omitted.

I'm building a node.js application now and WISH we had something half as good as Sidekiq written in Javascript.

How would you say it compares to something like Celery?

I found sidekiq to be better than celery for my needs because it was better integrated (although less configurable). It did what it said on the box without a fuss and without me having to debate the merits of and choose a backend, etc...

It worked with redis and it worked well.

Re: Disque 1.0 RC1 is out

#30

Post by lobster_johnson » Sat, Jan 02, 2016, 10:47 PM UTC

BradRuderman wrote: post #11

vdfs wrote: post #8

Is there a product-ready alternative to Disque?

It seems like Disque is a "simplified" Kafka, or a more vertically purposed Redis. It seems that it differs from RabbitMQ significantly because rabbit requires a queue to push the messages in, while Disque allows jobs to be pushed independently of consumers being setup.

Kafka is arguably simpler, at least in terms of data model, than Disque. Disque needs to handle acking and queue mutation and things like lock timeouts and dead lettering, all of which complicates its queue structure and API. Kafka queues (partitions, to be precise) are append-only, which simplifies a lot of things.