21–30 of 41 posts
How is the performance, how good does it scale horizontal/vertical? Could it be used as a replacement to Kafka message queue/ring? (me is still looking for a Kafka-like piece coded in Go)
Otherwise, the cloud native landscape has some similar projects listed in the same category as kafka: https://github.com/cncf/landscape
> Beanstalk is a simple, fast work queue.
> Its interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously.
The protocol is easy to drive directly and there are good libraries for most common languages.
How is the performance, how good does it scale horizontal/vertical? Could it be used as a replacement to Kafka message queue/ring? (me is still looking for a Kafka-like piece coded in Go)
This is a cool idea, but having the worker use http is just asking for problems. What happens if the http connection times out? For a worker architecture, it can be expected for some jobs to take north of 5 minutes. Http will time out by then, but the worker will still keep on going.
I appreciate any entry into the space but this is a really crowded space. With some very battle tested entries in the field as people have pointed out.
There are a few questions not in the readme that I think need answering for a queue system:
- Execution guarantees (at least once, at most once, exactly once?)
- Order guarantees (FIFO, approximately FIFO, nondeturministic, etc)
- Throughput compared to other systems
- Fault tolerance characteristics, how many nodes can I lose before it stops working, when it does how do I recover
As I said, I have a lot of choices in this field. I'd like to see all the data up front.
This is a cool idea, but having the worker use http is just asking for problems. What happens if the http connection times out? For a worker architecture, it can be expected for some jobs to take north of 5 minutes. Http will time out by then, but the worker will still keep on going.
It would be interesting to hear the rationale behind using this over something like RabbitMQ, which has its own storage layer, as well as the queue aspect.
As other comments have pointed out, HTTP is not a great idea because timeouts, etc.
The one subtle benefit that I can relay is that by using your main database as the storage layer, you can enqueue tasks within a transaction, as per: https://github.com/thruflo/ntorque/blob/master/src/ntorque/c...