Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka at the low end: how bad can it get?

broot.ca

Post reply on HN
Print view View original

101–110 of 144 posts

Re: Kafka at the low end: how bad can it get?

#101

Post by gnfargbl »

akshayshah wrote: post #99
gnfargbl wrote: post #79

Earlier quoted context omitted.

Pulsar. Works extremely well as both a job queue and a data bus. We have been using it in this application for half a decade now with no serious issues. I don't understand why it doesn't get more popular attention.

Even StreamNative is effectively abandoning Pulsar and going all-in on the Kafka protocol. I can see the theoretical benefits of Pulsar, but it just doesn’t seem to have the ecosystem momentum to compete with the Kafka juggernaut.

The advantages of Pulsar are very much practical, at least for us. Without it we would have to manage two separate messaging systems.

I don't see any evidence of StreamNative abandoning Pulsar at this point. I do see a compatibility layer for the Kafka protocol. That's fine.

Re: Kafka at the low end: how bad can it get?

#102

Post by boruto »

wmfiv wrote: post #98
boruto wrote: post #92

Earlier quoted context omitted.

How could I solve the problem of in-order processing based on a key using skip locked? Basically all records having the key to be processed one after other.

Work jobs in the order they were submitted within a partition key. This selects the next partition key that isn't locked. You could make it smarter to select a subset of the jobs checking for partition keys where all of the rows are still unlocked. SELECT * FROM jobs WHERE partition_key = ( SELECT partition_key FROM jobs ORDER BY partition_key LIMIT 1 SKIP LOCKED ) ORDER BY submitted_at FOR UPDATE SKIP LOCKED;

Yes, something along the lines could work. But I am not sure if the above query itself would work if rows are appended to the table in parallel.

Also if events for a partition gets processed quick would the last partition get an equal chance?

Re: Kafka at the low end: how bad can it get?

#103

Post by nickzelei »

ha-shine wrote: post #12
jszymborski wrote: post #3

What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.

I use Temporal ( https://temporal.io/ )

We also use temporal, it’s pretty great. Their ui makes it easy to debug workflows too.

Re: Kafka at the low end: how bad can it get?

#104

Post by kod »

alexwebr wrote: post #89
techcode wrote: post #84

What that post describes (all work going to one/few workers) in practice doesn't really happen if you properly randomize (e.g. just use random UUID) ID of the item/task when inserting it into Kafka. With that (and sharding based on that ID/value) - all your consumers/workers will get equal amount of messages/tasks. Both post and seemingly general theme of comments here is trashing choice of Kafka for low volume. Inte…

(Author here) You say: > What that post describes (all work going to one/few workers) in practice doesn't really happen if you properly randomize (e.g. just use random UUID) ID of the item/task when inserting it into Kafka. I would love to be wrong about this, but I don't _think_ this changes things. When you have few enough messages, you can still get unlucky and randomly choose the "wrong" partitions. To me, it's a…

If it's a fundamental probability thing with randomized partition selection, put the actual probability of what you're describing in the article.

.25^20 is not a "somewhat unlucky sequence of events"

Re: Kafka at the low end: how bad can it get?

#105

Post by kod »

araes wrote: post #34

Having never actually used this platform before, does anybody know why they named it Kafka, with all the horrible meanings? Per Wiktionary, Kafkaesque: [1] 1. "Marked by a senseless, disorienting, often menacing complexity." 2. "Marked by surreal distortion and often a sense of looming danger." 3. "In the manner of something written by Franz Kafka." (like the software language was written by Franz Kafka) Example: Met…

Kafka wrote a lot, and destroyed most of what he wrote.

Seems like a good name for a high-volume distributed log that deletes based on retention, not after consumption.

Re: Kafka at the low end: how bad can it get?

#106

Post by akshayshah »

gnfargbl wrote: post #101
akshayshah wrote: post #99

Earlier quoted context omitted.

Even StreamNative is effectively abandoning Pulsar and going all-in on the Kafka protocol. I can see the theoretical benefits of Pulsar, but it just doesn’t seem to have the ecosystem momentum to compete with the Kafka juggernaut.

The advantages of Pulsar are very much practical, at least for us. Without it we would have to manage two separate messaging systems. I don't see any evidence of StreamNative abandoning Pulsar at this point. I do see a compatibility layer for the Kafka protocol. That's fine.

It sure looks like they’re going quite a ways beyond Kafka-on-Pulsar - the Ursa/Oxia work they’re focused on right now replaces BookKeeper and seems very firmly Kafka-oriented. Or does Ursa also work with the Pulsar protocol?

Re: Kafka at the low end: how bad can it get?

#107

Post by srhtftw »

monksy wrote: post #16
stickfigure wrote: post #9

Earlier quoted context omitted.

Until you hit scale, the database you're already using is fine. If that's Postgres, look up SELECT FOR UPDATE SKIP LOCKED. The major convenience here - aside from operational simplicity - is transactional task enqueueing. For hosted, SQS or Google Cloud Tasks. Google's approach is push-based (as opposed to pull-based) and is far and above easier to use than any other queueing system.

Famious last words. There are database as a queue antipattern warnings about this.

> Famious last words.

These weren't his last words, but Jim Gray had this to say about this so-called "antipattern".

Queues Are Databases (1995)

Message-oriented-middleware (MOM) has become an small industry. MOM offers queued transaction processing as an advance over pure client-server transaction processing. This note makes four points: Queued transaction processing is less general than direct transaction processing. Queued systems are built on top of direct systems. You cannot build a direct system atop a queued system. It is difficult to build direct, conversational, or distributed transactions atop a queued system. Queues are interesting databases with interesting concurrency control. It is best to build these mechanisms into a standard database system so other applications can use these interesting features. Queue systems need DBMS functionality. Queues need security, configuration, performance monitoring, recovery, and reorganization utilities. Database systems already have these features. A full-function MOM system duplicates these database features. Queue managers are simple TP-monitors managing server pools driven by queues. Database systems are encompassing many server pool features as they evolve to TP-lite systems.

https://arxiv.org/abs/cs/0701158

Re: Kafka at the low end: how bad can it get?

#108

Post by nobleach »

stickfigure wrote: post #94
nobleach wrote: post #87

Earlier quoted context omitted.

There's a wonderful Kafka Children's book that I always suggest every team I work with read: https://www.gentlydownthe.stream/ The way I describe Kafka is, "an event has transpired... sometimes you care, and choose to take an action based on that event" The way I describe RabbitMQ is, "there's a new ticket in the lineup... it needs to be grabbed for action or left in the lineup... or discarded" Definitely not perfect…

Your two-sentence description is excellent. That book, not so much.

I suppose that's fair.

Re: Kafka at the low end: how bad can it get?

#109

Post by alexwebr »

kod wrote: post #104
alexwebr wrote: post #89

Earlier quoted context omitted.

(Author here) You say: > What that post describes (all work going to one/few workers) in practice doesn't really happen if you properly randomize (e.g. just use random UUID) ID of the item/task when inserting it into Kafka. I would love to be wrong about this, but I don't _think_ this changes things. When you have few enough messages, you can still get unlucky and randomly choose the "wrong" partitions. To me, it's a…

If it's a fundamental probability thing with randomized partition selection, put the actual probability of what you're describing in the article. .25^20 is not a "somewhat unlucky sequence of events"

(Author here)

Fair enough. I agree .25^20 is basically infinitesimal, and even with a smaller exponent (like .25^3) the odds are not great, so I appreciate you calling this out.

Flipping this around, though, if you have 4 workers total and 3 are busy with jobs (1 idle), your next job has only a 25% chance of hitting the idle worker. This is what I see the most in practice; there is a backlog, and not all workers are busy even though there is a backlog.

Re: Kafka at the low end: how bad can it get?

#110

Post by Joel_Mckay »

javaunsafe2019 wrote: post #82
Joel_Mckay wrote: post #74

Earlier quoted context omitted.

I like working with folks that know a good pint, and value workmanship. If you are inferring someone writing software for several decades might share, than one might want to at least reconsider civility over ones ego. Best of luck =3

Neither being German or old are bad values from my point of view. But you tried a bit hard to flex with your past experiences tbh...

Many NDA do not really ever expire on some projects, most work is super boring, and recovering dysfunctional architectures with a well known piece of free community software is hardly grandstanding.

"It works! so don't worry about spending a day or two exploring..." should be the takeaway insight about Erlang/RabbitMQ. Have a wonderful day. =3

Post reply on HN