Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From Kafka to ZeroMQ for real-time log aggregation

tomasz.janczuk.org

Post reply on HN
Print view View original

31–40 of 110 posts

Re: From Kafka to ZeroMQ for real-time log aggregation

#31

Post by kal31dic »

PieterH wrote: post #19
bdowling wrote: post #6

But why ZeroMQ and not nanomsg?

See http://hintjens.com/blog:112 for my opinion on why nano isn't (wasn't, perhaps, as it seems to be doing better) a good choice.

With sincerely the greatest respect and admiration for what you achieved with ZeroMQ, Pieter, I think perhaps one might be a bit more nuanced when one isn't a neutral party. Full disclosure - I wrote a D wrapper, but I am not involved in nanomsg development and just a user.

There was some drama when the maintainer quit briefly before rejoining. Since then the gitter channel has been more active than I remember it being before. The mailing list is quiet it is true. Somebody just released a Rust version, and version 1.0.0 of nanomsg was indeed released.

You can see commit history here: https://github.com/nanomsg/nanomsg/commits/master/src

Re: From Kafka to ZeroMQ for real-time log aggregation

#32

Post by PieterH »

kal31dic wrote: post #31
PieterH wrote: post #19

Earlier quoted context omitted.

See http://hintjens.com/blog:112 for my opinion on why nano isn't (wasn't, perhaps, as it seems to be doing better) a good choice.

With sincerely the greatest respect and admiration for what you achieved with ZeroMQ, Pieter, I think perhaps one might be a bit more nuanced when one isn't a neutral party. Full disclosure - I wrote a D wrapper, but I am not involved in nanomsg development and just a user. There was some drama when the maintainer quit briefly before rejoining. Since then the gitter channel has been more active than I remember it bei…

Thanks for the updates. I've edited my comment. I've always wanted nano to succeed, just disliked the negative attitude to ZeroMQ expressed in its docs, which seemed unnecessary and damaging.

Re: From Kafka to ZeroMQ for real-time log aggregation

#33

Post by kaleidic »

kal31dic wrote: post #31
PieterH wrote: post #19

Earlier quoted context omitted.

See http://hintjens.com/blog:112 for my opinion on why nano isn't (wasn't, perhaps, as it seems to be doing better) a good choice.

With sincerely the greatest respect and admiration for what you achieved with ZeroMQ, Pieter, I think perhaps one might be a bit more nuanced when one isn't a neutral party. Full disclosure - I wrote a D wrapper, but I am not involved in nanomsg development and just a user. There was some drama when the maintainer quit briefly before rejoining. Since then the gitter channel has been more active than I remember it bei…

Note that I replied to a different version of the parent comment. Thanks for updating.

Personally I started using zeromq, but was a bit disturbed by the experience of the saltstack guys with timeout problems and memory leaks. Probably fixed now, I should think. But more than that, the LGPL and MPL 2.0 licences aren't great for a commercial project that might in future need to be distributed to a different entity. So the license alone is a reason one might welcome greater diversity in available choices.

Re: From Kafka to ZeroMQ for real-time log aggregation

#34

Post by kaleidic »

PieterH wrote: post #32
kal31dic wrote: post #31

Earlier quoted context omitted.

With sincerely the greatest respect and admiration for what you achieved with ZeroMQ, Pieter, I think perhaps one might be a bit more nuanced when one isn't a neutral party. Full disclosure - I wrote a D wrapper, but I am not involved in nanomsg development and just a user. There was some drama when the maintainer quit briefly before rejoining. Since then the gitter channel has been more active than I remember it bei…

Thanks for the updates. I've edited my comment. I've always wanted nano to succeed, just disliked the negative attitude to ZeroMQ expressed in its docs, which seemed unnecessary and damaging.

Yes, well the little guy always wants to unseat the dominant player, and when there is some personal history involved, things get more mixed up (technical, emotional, manner of expression). In your shoes I would be irritated by that too. But a project if it develops eventually transcends the personalities involved, and that seems to be happening now.

Re: From Kafka to ZeroMQ for real-time log aggregation

#35

Post by bachback »

kchoudhu wrote: post #28
bachback wrote: post #24

With ZeroMQ I had the worst possible results and experience. Honestly much of what it claims is bogus. It is highly optimized for certain cases and utterly useless for distributed systems. Try and find out in PUB/SUB what the IP addresses of the subscribers are. Not possible. In many cases you will be much better off learning TCP/IP yourself. In the mentioned case you simply iterate over the vector of subscribers - m…

The ZeroMQ documentation is pretty up front about the need for you to build those pieces yourself. It would appear you chose the wrong tool for your requirements.

ZeroMQ wants to be a neutral wrapper in any language but in the end its a C++ library enforcing C++ concepts. You can't map in a straightforward way process concepts from C++ to other languages (also the OS and VM sits in between that). In the end its just mapping programming logic to state machines. There are much better ways to do this and end up with something much more powerful (with first class meta-programming).

Re: From Kafka to ZeroMQ for real-time log aggregation

#37

Post by bachback »

PieterH wrote: post #27
bachback wrote: post #24

With ZeroMQ I had the worst possible results and experience. Honestly much of what it claims is bogus. It is highly optimized for certain cases and utterly useless for distributed systems. Try and find out in PUB/SUB what the IP addresses of the subscribers are. Not possible. In many cases you will be much better off learning TCP/IP yourself. In the mentioned case you simply iterate over the vector of subscribers - m…

If you try to use ZeroMQ to replace TCP with the same semantics, then no, it won't work.

How should ZeroMQ replace TCP? Internet runs on TCP/IP (last time I checked). In private networks many problems of public networks don't appear. I'm aware that organisations use it for their internal systems. As soon as one bridges to the outside world, one is going to hit with a problem.

Re: From Kafka to ZeroMQ for real-time log aggregation

#38

Post by PieterH »

kaleidic wrote: post #34
PieterH wrote: post #32

Earlier quoted context omitted.

Thanks for the updates. I've edited my comment. I've always wanted nano to succeed, just disliked the negative attitude to ZeroMQ expressed in its docs, which seemed unnecessary and damaging.

Yes, well the little guy always wants to unseat the dominant player, and when there is some personal history involved, things get more mixed up (technical, emotional, manner of expression). In your shoes I would be irritated by that too. But a project if it develops eventually transcends the personalities involved, and that seems to be happening now.

ZeroMQ is hardly dominant. It's a small player in a huge market and there was and is space for many more projects in this area. I'm glad nano seems healthy again, yet it's not enough, and I'll explain why.

In the end the whole point of ZeroMQ was to build new protocols and APIs for decentralized messaging. My real disappointment with nano was that it made zero effort to build on existing work (mainly, ZMTP) and instead just started again, as if thousands of people hadn't spent years figuring out what a decentralized messaging protocol might look like.

It was worse than that, in fact. Nano launched itself on a wave of negativity. It makes good press, and poor everything else. Such hate for one's own history and knowledge base isn't healthy. If a messaging product isn't aiming at interoperability as a primary goal, it is worthless.

I'm not a fan of the "IETF or bust" approach either. That just doesn't work if you're a decentralized community. We needed and still need lightweight processes for RFC development. We use such a process (Digistan's COSS) in our RFCs. It wasn't random. I built Digistan and COSS over years after seeing AMQP swallowed up and destroyed by a committee. Why isn't nano using a process like COSS?

Without interoperable protocols, all we have is a bunch of software projects. And they die. And then all this is for nothing and the proprietary systems will rule the world and our dreams of making distributed software cheap again will die as well.

And this makes me angry: nano had the chance to push this forwards, and threw it away like old trash. What a stupid, petty waste of opportunity and goodwill.

Re: From Kafka to ZeroMQ for real-time log aggregation

#39

Post by PieterH »

kaleidic wrote: post #33
kal31dic wrote: post #31

Earlier quoted context omitted.

With sincerely the greatest respect and admiration for what you achieved with ZeroMQ, Pieter, I think perhaps one might be a bit more nuanced when one isn't a neutral party. Full disclosure - I wrote a D wrapper, but I am not involved in nanomsg development and just a user. There was some drama when the maintainer quit briefly before rejoining. Since then the gitter channel has been more active than I remember it bei…

Note that I replied to a different version of the parent comment. Thanks for updating. Personally I started using zeromq, but was a bit disturbed by the experience of the saltstack guys with timeout problems and memory leaks. Probably fixed now, I should think. But more than that, the LGPL and MPL 2.0 licences aren't great for a commercial project that might in future need to be distributed to a different entity. So…

We use the LGPL license with a clause that makes it harmless. The MPLv2 license is not an issue. All complaints from businesses about licensing come straight to me, as they have for years. MPLv2 raises zero complaints.

ZeroMQ "memory leaks and timeouts" wasn't the issue. Do you have a reference for that?

Salt's issues with ZMTP were performance; they wanted to multicast securely, and ZMTP does not support that. I'd complain once again that making their own custom protocol (actually, it seems to be largely just CurveCP) rather than working with us to extend ZMTP was poor form. Also, Salt wanted a pure Python stack.

Re: From Kafka to ZeroMQ for real-time log aggregation

#40

Post by PieterH »

bachback wrote: post #37
PieterH wrote: post #27

Earlier quoted context omitted.

If you try to use ZeroMQ to replace TCP with the same semantics, then no, it won't work.

How should ZeroMQ replace TCP? Internet runs on TCP/IP (last time I checked). In private networks many problems of public networks don't appear. I'm aware that organisations use it for their internal systems. As soon as one bridges to the outside world, one is going to hit with a problem.

ZeroMQ is a messaging layer that runs over numerous network protocols (TCP, UDP, PGM, TIPC,...), as well as inter-process and inter-thread. It provides abstractions (like pub-sub) that work over all these protocols, and it shows the developer the abstraction using a socket-like API.

In practice you build services which are internally composed and scaled using threads and processes, and which talk to each other over plain (unencrypted) protocols, e.g. TCP between boxes, IPC between processes on the same box. And when you need to talk to the outside world you build bridges that can speak any protocol you like, such as HTTP, or ZeroMQ's encrypted TCP protocol (CurveZMQ). From the developer's point of view, it's mostly transparent. Obviously you need security infrastructure, e.g. key/certificate management.

You can build entire ZeroMQ applications using only encrypted TCP and they will run on public infrastructure. You can develop and test the same apps on a laptop. You might run the same code using IPC on a laptop, and CurveZMQ over the Internet.

So though ZeroMQ definitely uses TCP as one of its transport protocols, it's not a replacement for it.

Which means that doing things like asking for the IP address of a peer make no sense. What's the IP address of a thread, or another process on the same box? If you need identity information, you should not use the transport protocol for that.

Post reply on HN