Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Pulsar is an open-source distributed pub-sub messaging system

pulsar.apache.org

Post reply on HN
Print view View original

241–249 of 249 posts

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#241

Post by manigandham »

GordonS wrote: post #240
manigandham wrote: post #234

Earlier quoted context omitted.

Just because it's widely used doesn't mean it's good (see PHP). It's slow with single-threaded topics that usually max out around 25k msgs/sec, fragile with dropped connections, stalled queues, corrupted data, terrible clustering that breaks often and doesn't support sharding, has a silly HIPE mode where you can choose to compile the Erlang code for more performance which turns startup time into minutes, etc. It's po…

I didn't k ow about Redis pub/sub, I'll check it out! With a quick glance, I'm not sure it supports message durability (persisting messages to storage)? IME, any Erlang system is difficult for noobs to contribute to; Erlang's syntax alone is... frightening :) I really haven't hit any of the issues you mentioned with rabbit. Several clusters in production for years have been rock solid, connections are stable (and whe…

Redis pub/sub is ephemeral. Send to a topic and any active listeners will receive.

If you need persistence then Redis v5 has a new data type called Streams which is similar to Kafka/Pulsar. Push messages to a stream and read with multiple consumers (using consumer groups) that can ack each msg to track read state.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#242

Post by NicoJuicy »

GordonS wrote: post #225
NicoJuicy wrote: post #217

Earlier quoted context omitted.

Look up : outbox pattern

Was going to suggest this too. I'm build a distributed system with RabbitMQ just now, where producers may be offline due to transient networking issues. I write messages to a local SQLite database, with another thread responsible for sending them to RabbitMQ and deleting them on successful delivery.

Well, Postgres has an event source system.

NoSql for this seems to be a better use-case.

I also depend on nats.io instead of RabbitMQ

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#243

Post by GordonS »

NicoJuicy wrote: post #242
GordonS wrote: post #225

Earlier quoted context omitted.

Was going to suggest this too. I'm build a distributed system with RabbitMQ just now, where producers may be offline due to transient networking issues. I write messages to a local SQLite database, with another thread responsible for sending them to RabbitMQ and deleting them on successful delivery.

Well, Postgres has an event source system. NoSql for this seems to be a better use-case. I also depend on nats.io instead of RabbitMQ

Not sure what you're getting at with NoSql here - how does NoSql vs relational affect the choice of message broker/queue?

My (very limited, and purely gained through this thread!) understanding of nats suggests it doesn't handle persistence, instead leaving it up to producers and consumers - but I need to not lose any messages in the event of an outage. Waiting for consumers to signal completion of processing isn't practical, as it will sometimes take several minutes or even hours before messages are processed; I think the real solution here is persistence at the broker?

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#244

Post by NicoJuicy »

GordonS wrote: post #243
NicoJuicy wrote: post #242

Earlier quoted context omitted.

Well, Postgres has an event source system. NoSql for this seems to be a better use-case. I also depend on nats.io instead of RabbitMQ

Not sure what you're getting at with NoSql here - how does NoSql vs relational affect the choice of message broker/queue? My (very limited, and purely gained through this thread!) understanding of nats suggests it doesn't handle persistence, instead leaving it up to producers and consumers - but I need to not lose any messages in the event of an outage. Waiting for consumers to signal completion of processing isn't p…

I'm talking about the backend for storing the messages/events in the outbox pattern.

That's how to persist in case of a network failure.

The outbox-pattern pattern removes the send message from it's datastore when send/acknowledged.

Persistency is different from the outbox-pattern, it handles what is does if the message broker received the message. The outbox pattern handles failure before the message reached the message broker.

NoSql and event stores normally have methods to adjust data to the latest version of the model, eg. Postgress ( which can be used as NoSql using bson) supports js v8 transformations. This is useful for edge-cases in the outbox pattern.

In short, I was more or less talking about the tech used. Rabbit vs Nats and sqllite vs NoSql ( in my case, using Postgress as NoSql).

Ps. Nats can be persistent as others mentioned in this thread, look up: nats streaming

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#245

Post by mac01021 »

EdwardDiego wrote: post #182
throwawaysea wrote: post #157

A lot is said or referenced in this conversation about why people chose Pulsar over Kafka. I'm not an expert in this area but are there use cases where Kafka is still better?

As someone with a few years of Kafka and the ecosystem under my belt, but no experience of using Pulsar in anger, the areas I can see where Pulsar is behind are mainly ancillary, and will likely be caught up by the community given a year or two. Kafka Streaming - Pulsar functions don't intend to (by the looks of it) provide all of the functionality available in Kafka Streaming. E.g., joining streams, treating a strea…

Do they usually provide similar throughput on the same hardware?

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#246

Post by nvarsj »

squarecog wrote: post #126
nvarsj wrote: post #103

Earlier quoted context omitted.

It's an interesting observation. I think that the modern approach to distributed systems is moving towards golang style microservices and lightweight / simple system design with RPC communication, reconcile type loops for state reconciliation, and backing CP databases. I think this is the influence of k8s (and maybe google's approach to distributed systems). I will almost certainly get downvoted for this (as I always…

I understand your architecture criticism, and think it has merit, but I'm not sure why Apache gets dragged into that. Apache Airflow is in Python. Apache Arrow is in C. CouchDB is Erlang. There's a ton of projects Apache Foundation hosts that fit your description but it's a mistake, I think, to confuse individual projects with Apache in general. Bad enough that people confuse the license with the foundation.

Sorry, you are right. It would probably be more accurate to use Spring actually. I was thinking of Cassandra/Zookeeper specifically when I said Apache.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#247

Post by nvarsj »

james-mcelwain wrote: post #168
nvarsj wrote: post #103

Earlier quoted context omitted.

It's an interesting observation. I think that the modern approach to distributed systems is moving towards golang style microservices and lightweight / simple system design with RPC communication, reconcile type loops for state reconciliation, and backing CP databases. I think this is the influence of k8s (and maybe google's approach to distributed systems). I will almost certainly get downvoted for this (as I always…

> the JVM's awful GC This just makes it seem like you are trolling. JVM devs have done more to advance state of art in this area than any other language. The problem is that most JVM apps just produce too much garbage, not necessarily that the algo itself is awful. Either way, there's no such thing as an optimal GC algorithm, just different trade-offs depending on your use case. Not everyone cares about latency.

I stand by that - I really think the JVM has a bad GC, and it has cost businesses billions probably. There is no trade off here - the JVM & Java have incredibly high memory requirements due to poor design - and this leads to all kinds of issues (like the bad GC latencies). No other language has this kind of problem as badly as the JVM does, even other GC languages. And we shouldn't accept this anymore - look at how much better golang's GC is for real world usage.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#248

Post by EdwardDiego »

mac01021 wrote: post #245
EdwardDiego wrote: post #182

Earlier quoted context omitted.

As someone with a few years of Kafka and the ecosystem under my belt, but no experience of using Pulsar in anger, the areas I can see where Pulsar is behind are mainly ancillary, and will likely be caught up by the community given a year or two. Kafka Streaming - Pulsar functions don't intend to (by the looks of it) provide all of the functionality available in Kafka Streaming. E.g., joining streams, treating a strea…

Do they usually provide similar throughput on the same hardware?

I couldn't say tbh, I'm keen in running a trial with Pulsar alongside Kafka in production, might write it up when I've done so.

Re: Apache Pulsar is an open-source distributed pub-sub messaging system

#249

Post by ckdarby »

mattboyle wrote: post #91
ckdarby wrote: post #69

Earlier quoted context omitted.

> found the documentation very lacking Really? It is one of the few open source projects that we've felt has had modern documentation. How long ago was this? > As a small startup You'll spend more time & money on the OpEx cost with Kafka than picking up the client library for Pulsar.

It was about 6 months ago. I completely disagree with the opex of picking up kafka vs developing a whole client library. Please could you try and explain how you came to this conclusion?

I recently did a talk covering a lot of what I wrote: https://www.youtube.com/watch?v=jLruEmh3ve0
Post reply on HN