Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka at the low end: how bad can it get?

broot.ca

Post reply on HN
Print view View original

121–130 of 144 posts

Re: Kafka at the low end: how bad can it get?

#121

Post by munksbeer »

rockwotj wrote: post #120
drinker wrote: post #118

Earlier quoted context omitted.

Those are all good points and pros for redpanda vs Kafka but my question stills stands. Isn't redpanda designed for high-volume scale similar to the use cases for Kafka rather than the low volume workloads talked about in the article?

When the founder started it was designed to be two things: * easy to use * more efficient and lower latency than the big resources needed for Kafka The efficiency really matters at scale and low latency yes but the simplicity of deployment and use is also a huge win.

In kafka, if you require the highest durability for messages, you configure multiple nodes on different hosts, and probably data centres, and you require acks=all. I'd say this is the thing that pushes latency up, rather than the code execution of kafka itself.

How does redpanda compare under those constraints?

Re: Kafka at the low end: how bad can it get?

#122

Post by techcode »

alexwebr wrote: post #109
kod wrote: post #104

Earlier quoted context omitted.

If it's a fundamental probability thing with randomized partition selection, put the actual probability of what you're describing in the article. .25^20 is not a "somewhat unlucky sequence of events"

(Author here) Fair enough. I agree .25^20 is basically infinitesimal, and even with a smaller exponent (like .25^3) the odds are not great, so I appreciate you calling this out. Flipping this around, though, if you have 4 workers total and 3 are busy with jobs (1 idle), your next job has only a 25% chance of hitting the idle worker. This is what I see the most in practice; there is a backlog, and not all workers are…

With Kafka you normally don't pick a worker - Kafka does that. IIRC with some sort of consistent hashing - but for simplicity sake lets say it's just modulo 'messageID % numberOfShards'.

You control/configure numberOfShards - and its usually set to something order of magnitude bigger than your expected number of workers (to be precise - that's number of docker pods or hardware boxes/servers) - e.g. 32, 64 or 128.

So in practice - Kafka assigns multiple shards to each of your "workers" (if you have more workers than shards then some workers don't do any work).

And while each of your workers is limited to one thread for consuming Kafka messages. Each worker can still process multiple messages at the same time - in different async/threads.

Re: Kafka at the low end: how bad can it get?

#123

Post by kvakerok »

kevinherron wrote: post #32
kvakerok wrote: post #30

Earlier quoted context omitted.

You haven't seen the worst of it. We had to implement a whole kafka module for a SCADA system because Target already had unrelated kafka infrastructure. Instead of REST API or anything else sane (which was available), ultra low volume messaging is now done by JSON objects wrapped in kafka. Peak incompetence.

> for a SCADA system for Ignition?

Yep

Re: Kafka at the low end: how bad can it get?

#124

Post by techcode »

techcode wrote: post #122
alexwebr wrote: post #109

Earlier quoted context omitted.

(Author here) Fair enough. I agree .25^20 is basically infinitesimal, and even with a smaller exponent (like .25^3) the odds are not great, so I appreciate you calling this out. Flipping this around, though, if you have 4 workers total and 3 are busy with jobs (1 idle), your next job has only a 25% chance of hitting the idle worker. This is what I see the most in practice; there is a backlog, and not all workers are…

With Kafka you normally don't pick a worker - Kafka does that. IIRC with some sort of consistent hashing - but for simplicity sake lets say it's just modulo 'messageID % numberOfShards'. You control/configure numberOfShards - and its usually set to something order of magnitude bigger than your expected number of workers (to be precise - that's number of docker pods or hardware boxes/servers) - e.g. 32, 64 or 128. So…

To me it seems like your underlying assumptions is "1 worker can only work on one message/item at a time", right?

While you could also use Kafka like that - and it might even work for your use case, as long as you configure option (sorry forgot the name) that makes Kafka redistribute shards because particular workers/consumers are too slow.

AFAIK the usual way is for each worker to get more than one message/item at a time, and do the actual item/work in/through separate thread/work pool (or another async mechanism).

Kafka then keeps track of which messages were picked up by each worker/consumer, and how big is the gap between that and committed offset (marked as done).

It gets a bit more tricky if you: - can't afford to process some messages/work again (well at extreme end it might actually be a show stopper for using Kafka) - need to have automatic retry on error/fail, how quickly/slowly you want to retry, how many times to retry...etc. - can you afford to temporarily "lose" some pending (picked up from Kafka but offset not marked as done) items for random things (worker OOMKILLED, solar flare hit network cable ...)

We've actually solved some of these with simply having another (set of) worker(s) that consume same topic with a delay (imagine cron job that runs every 5 minutes). And doing things in case there's no record of task being done, putting it into same topic again for retry ...etc.

Re: Kafka at the low end: how bad can it get?

#125

Post by rockwotj »

munksbeer wrote: post #121
rockwotj wrote: post #120

Earlier quoted context omitted.

When the founder started it was designed to be two things: * easy to use * more efficient and lower latency than the big resources needed for Kafka The efficiency really matters at scale and low latency yes but the simplicity of deployment and use is also a huge win.

In kafka, if you require the highest durability for messages, you configure multiple nodes on different hosts, and probably data centres, and you require acks=all. I'd say this is the thing that pushes latency up, rather than the code execution of kafka itself. How does redpanda compare under those constraints?

Oh if you care about durability on Kafka vs Redpanda, see https://www.redpanda.com/blog/why-fsync-is-needed-for-data-s..., acks=all does not fsync (by default before acknowledging the write), so it's still not safe. We use raft for the data path, a proven replication protocol (not the custom ISR protocol) and fsync by default for safety (although if you're good with relaxed durability like in Kafka you can enable that too: https://www.redpanda.com/blog/write-caching-performance-benc...).

As for Redpanda vs Kafka in multi AZ setups and latency, the big win in Redpanda is tail latencies are kept low (we have a variety of techniques to do this). Here's some numbers here: https://www.redpanda.com/blog/kafka-kraft-vs-redpanda-perfor...

Multi AZ latency is mostly single digit millisecond (ref: https://www.bitsand.cloud/posts/cross-az-latencies/) and the JVM can easily take just as long during GC, which can drive up those tail latencies.

Re: Kafka at the low end: how bad can it get?

#126

Post by kvakerok »

Joel_Mckay wrote: post #110
javaunsafe2019 wrote: post #82

Earlier quoted context omitted.

Neither being German or old are bad values from my point of view. But you tried a bit hard to flex with your past experiences tbh...

Many NDA do not really ever expire on some projects, most work is super boring, and recovering dysfunctional architectures with a well known piece of free community software is hardly grandstanding. "It works! so don't worry about spending a day or two exploring..." should be the takeaway insight about Erlang/RabbitMQ. Have a wonderful day. =3

Coincidentally another SCADA module we made was handling bi-directional RabbitMQ comms. Not everyone is a one-trick pony :)

Re: Kafka at the low end: how bad can it get?

#127

Post by Joel_Mckay »

kvakerok wrote: post #126
Joel_Mckay wrote: post #110

Earlier quoted context omitted.

Many NDA do not really ever expire on some projects, most work is super boring, and recovering dysfunctional architectures with a well known piece of free community software is hardly grandstanding. "It works! so don't worry about spending a day or two exploring..." should be the takeaway insight about Erlang/RabbitMQ. Have a wonderful day. =3

Coincidentally another SCADA module we made was handling bi-directional RabbitMQ comms. Not everyone is a one-trick pony :)

With legacy equipment there is usually no such thing as a homogeneous ecosystem, as vendor industrial parts EOL all the time. Certainly room in the markets for better options with open protocols. =3

Re: Kafka at the low end: how bad can it get?

#128

Post by mensfeld »

film42 wrote: post #18
enether wrote: post #2

thankfully early access for KIP-932 is coming in 1-3 weeks as the 4.0.0 release gets published

First time I've heard of KIP-932 and it looks very good. The two biggest issues IMO are finding a good Kafka client in the language you need (even for ruby this is a challenge) and easy at-least-once workers. You can over partition and make at-least-once workers happen (if you have a good Kafka client), or you use an http gateway and give up safe at-least-once. Hopefully this will make it easier to build an at-least-…

Do you mind explaining what you mean by not being able to find a "good Kafka client" for Ruby? There are pretty good bindings to librdkafka and frameworks like Karafka (https://github.com/karafka/karafka/) that provide many functionalities, including a Web UI.

Re: Kafka at the low end: how bad can it get?

#129

Post by pritambaral »

maniacalhack0r wrote: post #60
pritambaral wrote: post #54

Earlier quoted context omitted.

I did sth similar. Designed and built for 10 million objects / hour. Picked up by workers in batches of 1k. Benchmark peaked above 200 million objects / hour with PG in a small VM. Fast forward two years, the curse of success strikes, and we have a much higher load than designed for. Redesigned to create batches on the fly and then `SELECT FOR UPDATE batch SKIP LOCKED LIMIT 1` instead of `SELECT FOR UPDATE object SKI…

smart. although, i guess that pushes the locking from selecting queue entries to making sure that objects are placed into exactly 1 batch. curious if you ran into any bottlenecks there?

> ... making sure that objects are placed into exactly 1 batch. curious if you ran into any bottlenecks there?

A single application-layer thread doing batches of batch creation (heh). Not instant, but fast enough. I did have to add 'batchmaker is done' onto the 'no batch left' condition for worker exit.

> ... that pushes the locking from selecting queue entries to ...

To selecting batches. A batch is immutable once created. If work has to be restarted to handle new/updated objects, all batches are wiped and the batchmaker (and workers, anyway) start over.

Re: Kafka at the low end: how bad can it get?

#130

Post by pritambaral »

winrid wrote: post #61
stickfigure wrote: post #58

Earlier quoted context omitted.

If that's true, you managed to do much better than these folks: https://softwaremill.com/mqperf/ Maybe you should write a letter?

It's possible the person you're replying to wasn't using replication, so it's entirely different. Those folks also used "synchronous_commit is set to remote_write" which will have a performance impact

This is correct. My use-case was safe with eventual consistency, so I could've even used `synchronous_commit=off`, but I kept it to 'local' to get a baseline. Was happy with the 60k number I got, so there was no need for 'off'.

But I think the biggest reason I hit that number so easily was the ridiculous ease of batching. Starting with a query to select one task at a time, "converting" it select multiple tasks instead is ... a change of a single integer literal. FOR UPDATE SKIP LOCKED works the same regardless of whether your LIMIT is 1 or 1000.

Post reply on HN