Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pulsar vs. Kafka

streamnative.io

Post reply on HN
Print view View original

31–40 of 103 posts

Re: Pulsar vs. Kafka

#31

Post by maps7 »

ramraj07 wrote: post #19
bsaul wrote: post #16

Earlier quoted context omitted.

For people currently interested in building an event based architecture, pulsar is definitely a very well known option.

I am squarely in the "people interested in building an event based architecture". Not a CS background, but know tech decently. I typically know the names of more of these apache projects than most people I've talked to IRL (though ostensibly I'm not part of the tech elite). Yet pulsar I only came across on HN a week back. And nothing changes due to the knowledge. It's yet another apache product which has great design…

> Yet pulsar I only came across on HN a week back.

Same - was it the comment saying nobody chooses Kafka anymore? I was surprised by it.

Re: Pulsar vs. Kafka

#32

Post by polote »

maps7 wrote: post #31
ramraj07 wrote: post #19

Earlier quoted context omitted.

I am squarely in the "people interested in building an event based architecture". Not a CS background, but know tech decently. I typically know the names of more of these apache projects than most people I've talked to IRL (though ostensibly I'm not part of the tech elite). Yet pulsar I only came across on HN a week back. And nothing changes due to the knowledge. It's yet another apache product which has great design…

> Yet pulsar I only came across on HN a week back. Same - was it the comment saying nobody chooses Kafka anymore? I was surprised by it.

If you search you will find that there are always the same 4-5 HN users who comment on pulsar/kafka topics arguing that Kafka is not a viable solution anymore. Even though almost everyone is still using Kafka

Re: Pulsar vs. Kafka

#33

Post by willvarfar »

doctor_eval wrote: post #25

> Lower end-to-end latency helps enterprises gain business insights faster. They lost me here. I can think of plenty of situations where reduced latency is beneficial, but not many situations where shaving a few milliseconds would make a difference to “business insight”! Although I suppose it is strictly correct, in the tautological sense...

So I've developed an in-house fancy thoroughbred 'real time' data warehouse, a very rare beast indeed, and its awesome.

Of course, our business is still running on nightly reports. But the tech is cool!

So I wanna say you're wrong, but I've got man-years invested in a system that hasn't been utilized to its full potential yet :(

Time will tell. Somewhere, some competitor will be using real-time insights to out-compete us.

Re: Pulsar vs. Kafka

#34

Post by bmcfeeley »

ramraj07 wrote: post #19
bsaul wrote: post #16

Earlier quoted context omitted.

For people currently interested in building an event based architecture, pulsar is definitely a very well known option.

I am squarely in the "people interested in building an event based architecture". Not a CS background, but know tech decently. I typically know the names of more of these apache projects than most people I've talked to IRL (though ostensibly I'm not part of the tech elite). Yet pulsar I only came across on HN a week back. And nothing changes due to the knowledge. It's yet another apache product which has great design…

I'm no expert, but my understanding is that pig is a combination of

- a language for specifying data transformations, and

- an engine to compile programs written in that language into mapreduce jobs to execute on a hadoop cluster

it was designed to easily map some common functional and SQL idioms (e.g. filter, group by w/ aggregation functions) to parallel execution for processing huge amounts of data.

Impala is another big data project that is an engine for planning and executing SQL on data stored in a hadoop cluster.

Zookeeper is... black magic??

Re: Pulsar vs. Kafka

#35

Post by sciurus »

For more details on pulsar and comparisons to similar systems see this series of blog posts from a rabbitmq developer.

https://jack-vanlightly.com/blog/2018/10/2/understanding-how...

https://jack-vanlightly.com/blog/2018/10/21/how-to-not-lose-...

https://jack-vanlightly.com/blog/2018/10/25/testing-producer...

https://jack-vanlightly.com/blog/2019/9/4/a-look-at-multi-to...

Re: Pulsar vs. Kafka

#36

Post by z9e »

polote wrote: post #32
maps7 wrote: post #31

Earlier quoted context omitted.

> Yet pulsar I only came across on HN a week back. Same - was it the comment saying nobody chooses Kafka anymore? I was surprised by it.

If you search you will find that there are always the same 4-5 HN users who comment on pulsar/kafka topics arguing that Kafka is not a viable solution anymore. Even though almost everyone is still using Kafka

Meanwhile the Confluent Kafka conference seems to double in size year after year. And AWS chooses to offer Kafka as a service, rather than Pulsar. These " is dead / not viable anymore!" comments are pretty rampant in the Big Data space. Some are true though but it still makes me roll my eyes.

These people seem to be invested (quite literally) in Pulsar disrupting Kafka's position in the pub/sub & event streaming space. This isn't to say Pulsar can't someday be superior, though. Time will tell. But Kafka has such a massive lead and it fits the needs for most people who need to do data integration that I don't see the need for everyone to uproot their solutions in favor of Pulsar.

Re: Pulsar vs. Kafka

#37

Post by z9e »

akerro wrote: post #11
polote wrote: post #3

All those articles of kafka vs pulsar are always biased (this one is from a company selling pulsar). There are so many of them that I can't get an opinion on which one is good for what.

Pulsar is more flexible and fault-tolerant. For me the most important thing is client can request log queue starting from specific log by id, it has better retrying mechanisms for logs that failed to be processed. But it has absurdly bad documentation. I had to learn many things about Pulsar by downloading src of their java library and just reading the code. Documentation on starting bookie, zookie, pulsar, pulsar-pr…

> For me the most important thing is client can request log queue starting from specific log by id

Kafka clients can start at a particular offset ID within a partition.

Re: Pulsar vs. Kafka

#38

Post by sciurus »

ramraj07 wrote: post #19
bsaul wrote: post #16

Earlier quoted context omitted.

For people currently interested in building an event based architecture, pulsar is definitely a very well known option.

I am squarely in the "people interested in building an event based architecture". Not a CS background, but know tech decently. I typically know the names of more of these apache projects than most people I've talked to IRL (though ostensibly I'm not part of the tech elite). Yet pulsar I only came across on HN a week back. And nothing changes due to the knowledge. It's yet another apache product which has great design…

If you can't understand what Zookeeper is, I'd recommend reading Martin Kleppmann's book Designing Data-Intensive Applications (https://dataintensive.net/).

You don't need a CS degree to work in this field (I don't have one either!) but there are fundamental concepts you need to understand in order to make informed decisions when designing distributed systems.

Re: Pulsar vs. Kafka

#40

Post by SmooL »

Is there any sort of 'single node' version of these frameworks? I'm very interested in building event-driven solutions, but I don't need the scale offered by kakfa/pulsar, and I really don't want all the complexity. Is there any reason nobody has made a smaller, less distributed event-centric DB?
Post reply on HN