Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka at the low end: how bad can it get?

broot.ca

Post reply on HN
Print view View original

111–120 of 144 posts

Re: Kafka at the low end: how bad can it get?

#111

Post by jiaaro »

kitd wrote: post #83
jszymborski wrote: post #3

What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.

Kafka with a different partitioner would have worked fine. The problem was that the web workers loaded up the same partition. Randomising the chosen partition would have removed, or at least alleviated, the stated problem.

Random and round robin partitioning are the configurations being discussed.

The main point of the article is that low message volumes mean you can get unlucky and end up with idle workers when there is still work to be done

Re: Kafka at the low end: how bad can it get?

#112

Post by dude187 »

InDubioProRubio wrote: post #77
atmosx wrote: post #68

Earlier quoted context omitted.

Oh no! Let’s be real: teams come to the infra team asking for a queue system. They give their requirements, and you—like a responsible engineer—suggest a more capable queue to handle their needs more efficiently. But no, they want Kafka. Kafka, Kafka, Kafka. Fine. You (meaning an entire team) set up Kafka clusters across three environments, define SLIs, enforce SLOs, make sure everything is production-grade. Then you…

Then all the guys who requested that stuff quit

Well duh! They got a kafkaesque promotion using their upgraded resume!

Re: Kafka at the low end: how bad can it get?

#113

Post by VenturingVole »

gnfargbl wrote: post #79
jszymborski wrote: post #3

What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.

Pulsar. Works extremely well as both a job queue and a data bus. We have been using it in this application for half a decade now with no serious issues. I don't understand why it doesn't get more popular attention.

Pulsar vs Kafka was a significant lesson to me: The "best" technology isn't always the winner.

I put it in quotes because I'm a massive fan of Pulsar and addressing the shortcomings of Kafka. However, with regards to some choices at a former workplace: The broader existing support/integration ecosystem along with Confluent's commercial capabilities won out with regards to technology choices and I was forced to acquiesce.

A bit like Betamax vs VHS, albeit that one pre-dates me significantly.

Re: Kafka at the low end: how bad can it get?

#114

Post by lijok »

cheema33 wrote: post #95
NovemberWhiskey wrote: post #14

Kafka for small message volumes is one of those distinct resume-padding architectural vibes.

I needed to synchronize some tables between MS SQL Server and PostgreSQL. In the future we will need to add ClickHouse database to the mix. When I last looked, the recommended way to do this was to use Debezium w/Kafka. So that is why we use it. Data volume is low. If anybody knows of a simpler way to accomplish this, please do let me know.

We used a binlog reader library for Python, wrapped it in some 50 loc of rudimentary integration code and hosted it on some container somewhere.

Data volume was low though.

Re: Kafka at the low end: how bad can it get?

#115

Post by lmm »

xmcqdpt2 wrote: post #93
lmm wrote: post #29

Earlier quoted context omitted.

Kafka. If your load is low enough for the problem described in the article to happen, your load is low enough that it's not an issue.

I'm not sure you understood the article. You can have a very low load but each task on your queue takes a while to process, in which case you want fair distribution of work.

The distribution is fair - everything is round-robin, so in the long run each worker receives the same rate of tasks. It's just "lumpy" - sometimes you can get a big batch sent to one worker, then a big batch sent to another worker - but it will all average out.

Re: Kafka at the low end: how bad can it get?

#116

Post by drinker »

ofrzeta wrote: post #62
bassp wrote: post #49

Earlier quoted context omitted.

I use Kafka for a low-message-volume use case because it lets my downstream consumers replay messages… but yeah in most cases, it’s over kill

That was also a use case for me. However at some point I replaced Kafka with Redpanda.

Isn't redpanda built for the same scale requirements as Kafka?

Re: Kafka at the low end: how bad can it get?

#117

Post by rockwotj »

drinker wrote: post #116
ofrzeta wrote: post #62

Earlier quoted context omitted.

That was also a use case for me. However at some point I replaced Kafka with Redpanda.

Isn't redpanda built for the same scale requirements as Kafka?

Redpanda is much more lean and scales much better for low latency use cases. It does a bunch of kernel bypass and zero copy mechanisms to deliver low latency. Being in C++ means it can fit into much smaller footprints than Apache Kafka for a similar workload

Re: Kafka at the low end: how bad can it get?

#118

Post by drinker »

rockwotj wrote: post #117
drinker wrote: post #116

Earlier quoted context omitted.

Isn't redpanda built for the same scale requirements as Kafka?

Redpanda is much more lean and scales much better for low latency use cases. It does a bunch of kernel bypass and zero copy mechanisms to deliver low latency. Being in C++ means it can fit into much smaller footprints than Apache Kafka for a similar workload

Those are all good points and pros for redpanda vs Kafka but my question stills stands. Isn't redpanda designed for high-volume scale similar to the use cases for Kafka rather than the low volume workloads talked about in the article?

Re: Kafka at the low end: how bad can it get?

#119

Post by rockwotj »

stephen_g wrote: post #52
jszymborski wrote: post #3

What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.

Has anyone used Redpanda? I stumbled upon it when researching streaming, it claims to be Kafka compatible but higher performance and easier to manage. Haven't tried it myself but interested if anyone else has experience.

Plenty of people choose Redpanda because it’s the easiest getting started experience for kafka. There is a single binary for the full broker, where I have never seen Apache Kafka as easy to setup. It’s got a great UI as well.

The quickstart to getting everything running locally all documented here: https://docs.redpanda.com/current/get-started/quick-start/#d...

Disclaimer: I work at Redpanda

Re: Kafka at the low end: how bad can it get?

#120

Post by rockwotj »

drinker wrote: post #118
rockwotj wrote: post #117

Earlier quoted context omitted.

Redpanda is much more lean and scales much better for low latency use cases. It does a bunch of kernel bypass and zero copy mechanisms to deliver low latency. Being in C++ means it can fit into much smaller footprints than Apache Kafka for a similar workload

Those are all good points and pros for redpanda vs Kafka but my question stills stands. Isn't redpanda designed for high-volume scale similar to the use cases for Kafka rather than the low volume workloads talked about in the article?

When the founder started it was designed to be two things:

* easy to use * more efficient and lower latency than the big resources needed for Kafka

The efficiency really matters at scale and low latency yes but the simplicity of deployment and use is also a huge win.

Post reply on HN