Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka at the low end: how bad can it get?

broot.ca

Post reply on HN
Print view View original

11–20 of 144 posts

Re: Kafka at the low end: how bad can it get?

#11

Post by xyst »

> Each of these Web workers puts those 4 records onto 4 of the topic’s partitions in a round-robin fashion. And, because they do not coordinate this, they might choose the same 4 partitions, which happen to all land on a single consumer

Then choose a different partitioning strategy. Often key based partitioning can solve this issue. Worst case scenario, you use a custom partitioning strategy.

Additionally , why can’t you match the number of consumers in consumer group to number of partitions?

The KIP mentioned seems interesting though. Kafka folks trying to make a play towards replacing all of the distributed messaging systems out there. But does seem a bit complex on the consumer side, and probably a few foot guns here for newbies to Kafka. [1]

[1] https://cwiki.apache.org/confluence/plugins/servlet/mobile?c...

Re: Kafka at the low end: how bad can it get?

#13

Post by vrosas »

stickfigure wrote: post #9
jszymborski wrote: post #3

What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.

Until you hit scale, the database you're already using is fine. If that's Postgres, look up SELECT FOR UPDATE SKIP LOCKED. The major convenience here - aside from operational simplicity - is transactional task enqueueing. For hosted, SQS or Google Cloud Tasks. Google's approach is push-based (as opposed to pull-based) and is far and above easier to use than any other queueing system.

Cloud Tasks is one of the most undervalued tools in the GCP ecosystem, but mostly because PubSub gets all the attention. I've been using it since it was baked in the AppEngine and love it for 1-to-1 queues or delayed job handling.

Re: Kafka at the low end: how bad can it get?

#15

Post by ozarker »

jszymborski wrote: post #3

What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.

SQS, Azure Service Bus, RabbitMQ, ActiveMQ, QPID, etc… any message broker that provides the competing consumer pattern. though I’ll say having managed many of these message brokers myself, it’s definitely better paying for a managed service. They’re a nightmare when you start running into problems.

Re: Kafka at the low end: how bad can it get?

#16

Post by monksy »

stickfigure wrote: post #9
jszymborski wrote: post #3

What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.

Until you hit scale, the database you're already using is fine. If that's Postgres, look up SELECT FOR UPDATE SKIP LOCKED. The major convenience here - aside from operational simplicity - is transactional task enqueueing. For hosted, SQS or Google Cloud Tasks. Google's approach is push-based (as opposed to pull-based) and is far and above easier to use than any other queueing system.

Famious last words. There are database as a queue antipattern warnings about this.

Re: Kafka at the low end: how bad can it get?

#18

Post by film42 »

enether wrote: post #2

thankfully early access for KIP-932 is coming in 1-3 weeks as the 4.0.0 release gets published

First time I've heard of KIP-932 and it looks very good. The two biggest issues IMO are finding a good Kafka client in the language you need (even for ruby this is a challenge) and easy at-least-once workers.

You can over partition and make at-least-once workers happen (if you have a good Kafka client), or you use an http gateway and give up safe at-least-once. Hopefully this will make it easier to build an at-least-once style gateway that's easier to work with across a variety of languages. I know many have tried in the past but not dropping messages is hard to do right.

Re: Kafka at the low end: how bad can it get?

#19

Post by reval »

monksy wrote: post #16
stickfigure wrote: post #9

Earlier quoted context omitted.

Until you hit scale, the database you're already using is fine. If that's Postgres, look up SELECT FOR UPDATE SKIP LOCKED. The major convenience here - aside from operational simplicity - is transactional task enqueueing. For hosted, SQS or Google Cloud Tasks. Google's approach is push-based (as opposed to pull-based) and is far and above easier to use than any other queueing system.

Famious last words. There are database as a queue antipattern warnings about this.

Why is that an anti-pattern? Databases have added `SKIP LOCKED` and `SELECT FOR UPDATE` to handle these use cases. What are the downsides?

Re: Kafka at the low end: how bad can it get?

#20

Post by sanex »

ozarker wrote: post #15
jszymborski wrote: post #3

What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.

SQS, Azure Service Bus, RabbitMQ, ActiveMQ, QPID, etc… any message broker that provides the competing consumer pattern. though I’ll say having managed many of these message brokers myself, it’s definitely better paying for a managed service. They’re a nightmare when you start running into problems.

If you're using .NET I have to plug https://particular.net/ Nservicebus from particular.net. It's great at abstracting away the underlying message broker and provides an opinionated way to build a distributed system.
Post reply on HN