Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An introduction to RabbitMQ

erlang-solutions.com

Post reply on HN
Print view View original

221–230 of 263 posts

Re: An introduction to RabbitMQ

#221

Post by lbruder »

kabacha wrote: post #212
formerlurker2 wrote: post #180

Earlier quoted context omitted.

I'd be interested to know what the 3 lines of python were and also more details about the redis server you deployed to replace RabbitMQ during the outage.

I'm using a bit of a hyperbole but using redis list as a task queue is a simple as: while True: msg = r.lpop(key) try: result = do_something(msg) except Exception as e: log.error(f'failure for msg "{msg}" got {e} back to queue {key}') r.rpush(key, msg) continue Pop a member from a list, if failed plop it back to the end. It's simple, explicit and just works™

It just works, until PUSHing the member back fails. At least use RPOPLPUSH and a separate worker queue to make sure you don't accidentally drop packets.

Don't get me wrong, I'm using Redis queues myself and trying to get rid of the last remnants of RabbitMQ in our code, but there are use cases where RabbitMQ means less thinking about stuff, because it just works(tm)...

Re: An introduction to RabbitMQ

#222

Post by elisharobinson »

kabacha wrote: post #177

I'm really surprised to see so much positivity about rabbitmq here when it's probably the most sweared-at software in the space. Let me share my anecdote. In my last work place I got onboarded on rabbitmq and it was such a painful software to work with and almost impossible to set up locally that I silently sneaked in simple redis list as queue alternative for my dev environment. The whole rabbitmq and it's pika libr…

Your fault is using pika. I thought the same untill I tried some other python libraries . Try Oslo messaging . It's used in openstack (enough said).

Re: An introduction to RabbitMQ

#223

Post by st1ck »

kabacha wrote: post #177

I'm really surprised to see so much positivity about rabbitmq here when it's probably the most sweared-at software in the space. Let me share my anecdote. In my last work place I got onboarded on rabbitmq and it was such a painful software to work with and almost impossible to set up locally that I silently sneaked in simple redis list as queue alternative for my dev environment. The whole rabbitmq and it's pika libr…

Does your solution handle the situation when consumer crashes and queue has to be accumulated (while RAM allows) until consumer is up again (maybe 1 day later)? AFAIK ZMQ can't guarantee this.

Re: An introduction to RabbitMQ

#224

Post by twic »

st1ck wrote: post #223
kabacha wrote: post #177

I'm really surprised to see so much positivity about rabbitmq here when it's probably the most sweared-at software in the space. Let me share my anecdote. In my last work place I got onboarded on rabbitmq and it was such a painful software to work with and almost impossible to set up locally that I silently sneaked in simple redis list as queue alternative for my dev environment. The whole rabbitmq and it's pika libr…

Does your solution handle the situation when consumer crashes and queue has to be accumulated (while RAM allows) until consumer is up again (maybe 1 day later)? AFAIK ZMQ can't guarantee this.

It must be emphasised that, despite the name, ZeroMQ is not a message queue. It is a networking library. The current blurb says:

> ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework.

And the old meme was "ZeroMQ is a replacement for Berkeley sockets".

It's a pretty cool networking library. But it makes no sense to think of it in the same slot as RabbitMQ, or even Redis.

That the GP mentioned it does make me wonder if they don't really understand what they're doing.

EDIT I love that the official guide actually has this diagram in! Pieter Hintjens's death was a sad loss. http://zguide.zeromq.org/page:all#How-It-Began

Re: An introduction to RabbitMQ

#226

Post by ayush--s »

st1ck wrote: post #223
kabacha wrote: post #177

I'm really surprised to see so much positivity about rabbitmq here when it's probably the most sweared-at software in the space. Let me share my anecdote. In my last work place I got onboarded on rabbitmq and it was such a painful software to work with and almost impossible to set up locally that I silently sneaked in simple redis list as queue alternative for my dev environment. The whole rabbitmq and it's pika libr…

Does your solution handle the situation when consumer crashes and queue has to be accumulated (while RAM allows) until consumer is up again (maybe 1 day later)? AFAIK ZMQ can't guarantee this.

a Redis list (what he's using) would handle this easily (I think Redis' PubSub also handles this)

Re: An introduction to RabbitMQ

#227

Post by ayush--s »

kabacha wrote: post #177

I'm really surprised to see so much positivity about rabbitmq here when it's probably the most sweared-at software in the space. Let me share my anecdote. In my last work place I got onboarded on rabbitmq and it was such a painful software to work with and almost impossible to set up locally that I silently sneaked in simple redis list as queue alternative for my dev environment. The whole rabbitmq and it's pika libr…

rabbitmq used to be one of the hardest things to self-host (even at relatively small scale). But it has been pretty stable since moving to Cloudamqp as a managed solution.

Re: An introduction to RabbitMQ

#228

Post by jen20 »

twic wrote: post #224
st1ck wrote: post #223

Earlier quoted context omitted.

Does your solution handle the situation when consumer crashes and queue has to be accumulated (while RAM allows) until consumer is up again (maybe 1 day later)? AFAIK ZMQ can't guarantee this.

It must be emphasised that, despite the name, ZeroMQ is not a message queue. It is a networking library. The current blurb says: > ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. And the old meme was "ZeroMQ is a replacement for Berkeley sockets". It's a pretty cool networking library. But it makes no sense to think of it in the same slot as R…

As an aside, the ZeroMQ Guide is the finest piece of technical writing I have ever seen - closely followed by Pieter’s (sadly unfinished) “Scalable C” book. His non-technical writing is also excellent.

Re: An introduction to RabbitMQ

#229

Post by ayush--s »

kabacha wrote: post #212
formerlurker2 wrote: post #180

Earlier quoted context omitted.

I'd be interested to know what the 3 lines of python were and also more details about the redis server you deployed to replace RabbitMQ during the outage.

I'm using a bit of a hyperbole but using redis list as a task queue is a simple as: while True: msg = r.lpop(key) try: result = do_something(msg) except Exception as e: log.error(f'failure for msg "{msg}" got {e} back to queue {key}') r.rpush(key, msg) continue Pop a member from a list, if failed plop it back to the end. It's simple, explicit and just works™

couple of problems with this: (1) RPUSH can fail (2) LPOP's result might never reach your client after redis has executed it.

Re: An introduction to RabbitMQ

#230

Post by ayush--s »

webscalist wrote: post #117

RabbitMQ has huge learning curve if you're trying to build a worker queue. First, you'll learn about ack/noack and get the worker ack on success. Then, you'll learn about dead letter queue ... etc for delayed retries. Now, you'll have a topic exchange and a bit hairy routing in place using wildcards. And you mistakenly set dead letter routing key so that expired messages end up in multiple queues (retry queues and ac…

isn't Rabbitmq prone to "split brain" problem on HA setups?
Post reply on HN