71–80 of 110 posts
Earlier quoted context omitted.
This would be great, but I don't think it works on AWS -- I don't think they support broadcast.
My ZeroMQ components all register themselves in a database when coming and going. This makes it trivially easy to find where stuff is just by running a bog simple database query. Lots of ways to skin this cat...
I don't understand why people need such ridiculously fast systems when we are using RabbitMQ and crappy Apache flume and we generate more than 5k with spikes of 50k messages/second. Please author of the article tell me your metrics. And our log messages are ridiculously big at times (15k to as big as 50k). Our pipe never has problems. What fails for us is Elastic Search. In fact at one point in the past we did 100k m…
We run an average of 14k logs/sec through a two-node RMQ cluster, with max sustained throughput in the ~50k range. You're spot on with the bottleneck being Elasticsearch, but the latest releases in the 2.x train have a lot of fine adjustments that have drastically improved our indexing rate, such that we actually index at a 50k/sec rate. Would be interested to hear about your ES cluster configuration.
http://wanderr.com/jay/tail-error-logs-to-slack-for-fun-and-...
I used to be on a team responsible for a single small-ish Kafka cluster (between 6-12 nodes) doing non-trivial throughput on bare metal. Without commenting on whether ZeroMQ is the right alternative: I can understand being scared off. Our hand was forced such that we had to go the other way and understand what was going on in Kafka. The kicker is that Kafka can be rock solid in terms of handling massive throughput an…
Kafka and Confluent Platform are very much still works in progress. I had to patch Kafka Connect HDFS connector because a fix I needed was left out of the last release. Be prepared to do something similar with any of Kafka's components.
Earlier quoted context omitted.
My ZeroMQ components all register themselves in a database when coming and going. This makes it trivially easy to find where stuff is just by running a bog simple database query. Lots of ways to skin this cat...
What if they crash without registering an exit?
Earlier quoted context omitted.
My ZeroMQ components all register themselves in a database when coming and going. This makes it trivially easy to find where stuff is just by running a bog simple database query. Lots of ways to skin this cat...
indeed, with the small caveat that you now need to know the database's IP address, thereby introducing a failure point / master IP point. My solution is completely IP address agnostic. Indeed you could run my version even with DHCP for all nodes.
As for the SPOF: my logic was that if the DB is dead, there's no point having the ZeroMQ components come up anyhow. The jobs will keep on trying to come up, but until Ops brings the database back up, they can't do any damage.
Earlier quoted context omitted.
indeed, with the small caveat that you now need to know the database's IP address, thereby introducing a failure point / master IP point. My solution is completely IP address agnostic. Indeed you could run my version even with DHCP for all nodes.
For sure. In my case, the bootstrap IP is distributed during initial software configuration. Not elegant as your solution, but it works. As for the SPOF: my logic was that if the DB is dead, there's no point having the ZeroMQ components come up anyhow. The jobs will keep on trying to come up, but until Ops brings the database back up, they can't do any damage.
It's amazing how small use case differences can mean very big differences in the effectiveness of various strategies. The at least once / at most once / no guarantee either way but soft real time, use cases, can mean radically different toolchains. This is the real lesson of distributed streaming.
Earlier quoted context omitted.
My ZeroMQ components all register themselves in a database when coming and going. This makes it trivially easy to find where stuff is just by running a bog simple database query. Lots of ways to skin this cat...
What if they crash without registering an exit?
But why ZeroMQ and not nanomsg?
I knew a big part of the reliability problems we were having was related to the distributed state that needed to be kept synchronized. I wanted to move to something simpler that did not rely on any durable, distributed state, while supporting the messaging patters we required. ZeroMQ fit the bill.
While there were other implementations with similar properties, there is no reasonable way to compare them up front given that what makes the real difference at the end of the day is the behavior of the system at 2am one day after a prolonged stress run. As a startup one does not have resources to conduct an up front analysis of that sort. You just take a bet. If it does not pan out, you pivot. This is exactly what we have done with the move from Kafka to ZeroMQ in the first place.
Now that we've been using ZeroMQ for over a year and have been perfectly happy, there is no incentive to look elsewhere.