Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Disque – a distributed message broker

github.com

Post reply on HN
Print view View original

71–80 of 96 posts

Re: Disque – a distributed message broker

#71

Post by lumpypua »

jonny_eh wrote: post #65
pyre wrote: post #58

Earlier quoted context omitted.

It depends on the use-case. You may want the request to your web server to wait for a response from your transaction processor (e.g. Stripe) before returning.

That shouldn't be done in a web request. Not only can it lead to wasted time in the request handler, but it can lead to race conditions like double purchases.

How do you implement "no really, this request should not return until the server has successfully completed external communication"?

You can do some sort of websocket-y mechanism to tell the client that the server has completed a previous request, or polling, but both of those bring a lot of infrastructure overhead.

Re: Disque – a distributed message broker

#72

Post by pyre »

jonny_eh wrote: post #65
pyre wrote: post #58

Earlier quoted context omitted.

It depends on the use-case. You may want the request to your web server to wait for a response from your transaction processor (e.g. Stripe) before returning.

That shouldn't be done in a web request. Not only can it lead to wasted time in the request handler, but it can lead to race conditions like double purchases.

> it can lead to race conditions like double purchases

"Let's return as soon as possible rather than blocking" is not the real way to handle race conditions. All you are doing is reducing the surface (time) that such a thing can happen in. It makes more sense to plan these things out using something else (e.g. locking).

Re: Disque – a distributed message broker

#73

Post by jonny_eh »

lumpypua wrote: post #71
jonny_eh wrote: post #65

Earlier quoted context omitted.

That shouldn't be done in a web request. Not only can it lead to wasted time in the request handler, but it can lead to race conditions like double purchases.

How do you implement "no really, this request should not return until the server has successfully completed external communication"? You can do some sort of websocket-y mechanism to tell the client that the server has completed a previous request, or polling, but both of those bring a lot of infrastructure overhead.

Polling is super easy. It's just one more API endpoint.

Re: Disque – a distributed message broker

#74

Post by StavrosK »

jonny_eh wrote: post #73
lumpypua wrote: post #71

Earlier quoted context omitted.

How do you implement "no really, this request should not return until the server has successfully completed external communication"? You can do some sort of websocket-y mechanism to tell the client that the server has completed a previous request, or polling, but both of those bring a lot of infrastructure overhead.

Polling is super easy. It's just one more API endpoint.

Blocking is even easier, it's zero more endpoints.

Re: Disque – a distributed message broker

#75

Post by revskill »

jonny_eh wrote: post #42
antirez wrote: post #22

Earlier quoted context omitted.

Let's say you have a web app that sends emails, and you have many worker processes actually sending emails. If you use something like Disque, instead of sending the email directly from the web app, you send a message to Disque, into the send_email queue: "Please send an email to foo@example.com". Workers sending emails will fetch from the send_email queue, and will get the message. Workers sending emails are supposed…

Isn't Redis already good for this, powering backgrounding software like Resque and Sidekiw? What does Disque offer that's different? BTW, thanks for Redis. It's one of my favourite pieces of software ever. Edit: NVM, found this: http://antirez.com/news/88

If your Redis died, you must have to saved the sent messages to process later. It's the most important fact about distributed message queue.

Re: Disque – a distributed message broker

#77

Post by coleifer »

I would suggest that one of the reasons so many people use Redis as a message queue is because they do not want another piece of infrastructure. The fact that people use Redis as a message queue does not, in my mind, mean that this is for lack of proven alternatives. I think this is a cool project, but for me, I will stick with Redis.

Re: Disque – a distributed message broker

#78

Post by StavrosK »

cdelsolar wrote: post #76
StavrosK wrote: post #74

Earlier quoted context omitted.

Blocking is even easier, it's zero more endpoints.

Yeah and if Stripe goes down your server goes down with it.

How are you architecting things that an upstream error will cause your server to die?

Look, in both cases, the request needs to be synchronous. You need to tell the customer that their purchase went through. If you're going to add a whole other bus into the thing and make it asynchronous and have an elaborate polling solution on the client, that's fine, but I would (and do) just have them wait the two extra seconds and either say "your card has been charged" or handle the rare case with "there has been an upstream error, please don't close this window while we retry the request".

Re: Disque – a distributed message broker

#79

Post by thezilch »

coleifer wrote: post #77

I would suggest that one of the reasons so many people use Redis as a message queue is because they do not want another piece of infrastructure. The fact that people use Redis as a message queue does not, in my mind, mean that this is for lack of proven alternatives. I think this is a cool project, but for me, I will stick with Redis.

I think it's more than that other message queues are "another piece of infrastructure." In fact, many other message queues are several pieces of software, and while I can appreciate the modular-nature / separation of concerns, it can be very overwhelming to introduce, for example, JVM, Zookeeper, another DB, and the queuing software to handle the above. I'd much rather have a single, small, and yet still powerful piece of software like Redis, and if Disque can be that without the performance issues introduced by mapping a message queue over Redis, that seems to be a huge gain.

This is different enough from Redis that I don't see how they directly compete. Of course, if you already have a system setup and no performance constraints with what you have, of course you shouldn't change.

Post reply on HN