Live data from Hacker News

Pulsar vs. Kafka

streamnative.io

41–50 of 103 posts

Re: Pulsar vs. Kafka

#41
post #40

Is there any sort of 'single node' version of these frameworks? I'm very interested in building event-driven solutions, but I don't need the scale offered by kakfa/pulsar, and I really don't want all the complexity. Is there any reason nobody has made a smaller, less distributed event-centric DB?

Redis?

Re: Pulsar vs. Kafka

#42
post #40

Is there any sort of 'single node' version of these frameworks? I'm very interested in building event-driven solutions, but I don't need the scale offered by kakfa/pulsar, and I really don't want all the complexity. Is there any reason nobody has made a smaller, less distributed event-centric DB?

Redis Streams is a pretty good lightweight replacement for Kafka. Of course, what Kafka gets you are durability, being able to pick among at-least-once, at-most-once and exactly-once delivery of messages, among other things

Re: Pulsar vs. Kafka

#44
post #30

AWS should fork Pulsar and put out a v2 streaming product. Kinesis is kind of crappy (IMO) and doesn't seem to be improving much. If you look at the Pulsar architecture and feature set you can tell that it was designed very much with this in mind (something that large scale cloud providers can integrate with their infinitely scalable storage and compute systems). It's not all hype either, according to this post https…

Instead of Kinesis we you AWS MSK (Managed Kafka), which is expensive but works quite well.

Re: Pulsar vs. Kafka

#45
post #16

Earlier quoted context omitted.

For people currently interested in building an event based architecture, pulsar is definitely a very well known option.

I am squarely in the "people interested in building an event based architecture". Not a CS background, but know tech decently. I typically know the names of more of these apache projects than most people I've talked to IRL (though ostensibly I'm not part of the tech elite). Yet pulsar I only came across on HN a week back. And nothing changes due to the knowledge. It's yet another apache product which has great design…

Apache Pulsar should not be confused with another quite popular Pulsar:

Event driven concurrent framework for Python https://github.com/quantmind/pulsar

Re: Pulsar vs. Kafka

#46
post #40

Is there any sort of 'single node' version of these frameworks? I'm very interested in building event-driven solutions, but I don't need the scale offered by kakfa/pulsar, and I really don't want all the complexity. Is there any reason nobody has made a smaller, less distributed event-centric DB?

You can run kafka on a single node no problem.

Give this a try: https://www.digitalocean.com/community/tutorials/how-to-inst...

Re: Pulsar vs. Kafka

#47
post #40

Is there any sort of 'single node' version of these frameworks? I'm very interested in building event-driven solutions, but I don't need the scale offered by kakfa/pulsar, and I really don't want all the complexity. Is there any reason nobody has made a smaller, less distributed event-centric DB?

Postgres can be used as an effective event centric database and pub/sub queue with NOTIFY and LISTEN

Re: Pulsar vs. Kafka

#48
post #40

Is there any sort of 'single node' version of these frameworks? I'm very interested in building event-driven solutions, but I don't need the scale offered by kakfa/pulsar, and I really don't want all the complexity. Is there any reason nobody has made a smaller, less distributed event-centric DB?

You can run kafka on a single node no problem. Give this a try: https://www.digitalocean.com/community/tutorials/how-to-inst...

Oh cool, thank you!

Re: Pulsar vs. Kafka

#49
post #40

Is there any sort of 'single node' version of these frameworks? I'm very interested in building event-driven solutions, but I don't need the scale offered by kakfa/pulsar, and I really don't want all the complexity. Is there any reason nobody has made a smaller, less distributed event-centric DB?

Redis Streams is a pretty good lightweight replacement for Kafka. Of course, what Kafka gets you are durability, being able to pick among at-least-once, at-most-once and exactly-once delivery of messages, among other things

Thank you, I'll check it out

Re: Pulsar vs. Kafka

#50

How does it compare to https://docs.prefect.io/ ?

Prefect is workflow (particularly dataflow) orchestrator. Pulsar and Kafka are general purpose distributed streaming engines.

ah, yeah. totally forgot. sorry. i deal a lot with workflows and imagery mixes them up for me.
Post reply on HN