Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka at the low end: how bad can it get?

broot.ca

Post reply on HN
Print view View original

71–80 of 144 posts

Re: Kafka at the low end: how bad can it get?

#71

Post by kstrauser »

stickfigure wrote: post #53
maniacalhack0r wrote: post #35

Earlier quoted context omitted.

as with everything, it depends on how you're processing the queue. eg we built a system at my last company to process 150 million objects / hour, and we modeled this using a postgres-backed queue with multiple processes pulling from the queue. we observed that, whenever there were a lot of locked rows (ie lots of work being done), Postgres would correctly SKIP these rows, but having to iterate over and skip that many…

40,000 per second is waaaaay beyond where you should use a dedicated queuing solution. Even dedicated queues require tuning to handle that kind of throughput. (or you can just use SQS or google cloud tasks, which work out of the box)

I worked at a shop that had to process about 6M RPS for 5 seconds at a time, once a minute or so. That looked a lot like a boatload of Python background threads queueing work in memory then flushing them out into Cassandra. That was a fun little project.

Re: Kafka at the low end: how bad can it get?

#72

Post by voodooEntity »

We build an Infrastructure with about 6 microservices and Kafka as main message queue (job queue).

The problem the author describes is 100% true and if you are scaled with enaugh workers this can turn out really bad.

While not beeing the only issue we faced (others are more environment/project-language specific) we got to a point where we decided to switch from kafka to rabbitmq.

Re: Kafka at the low end: how bad can it get?

#73

Post by kyawzazaw »

atmosx wrote: post #68
NovemberWhiskey wrote: post #14

Kafka for small message volumes is one of those distinct resume-padding architectural vibes.

Oh no! Let’s be real: teams come to the infra team asking for a queue system. They give their requirements, and you—like a responsible engineer—suggest a more capable queue to handle their needs more efficiently. But no, they want Kafka. Kafka, Kafka, Kafka. Fine. You (meaning an entire team) set up Kafka clusters across three environments, define SLIs, enforce SLOs, make sure everything is production-grade. Then you…

We have way way way less than that in my team. But they don't support anything else.

Re: Kafka at the low end: how bad can it get?

#74

Post by Joel_Mckay »

javaunsafe2019 wrote: post #67
Joel_Mckay wrote: post #46

Earlier quoted context omitted.

We did something similar using RabbitMQ with bson over AMQP, and static message routing. Anecdotally, the design has been very reliable for over 6 years with very little maintenance on that part of the system, handles high-latency connection outage reconciliation, and new instances are cycled into service all the time. Mostly people that ruminate on naive choices like REST/HTTP2/MQTT will have zero clue how the probl…

I don’t know why but I could wear you are German (and old)

I like working with folks that know a good pint, and value workmanship.

If you are inferring someone writing software for several decades might share, than one might want to at least reconsider civility over ones ego. Best of luck =3

Re: Kafka at the low end: how bad can it get?

#75

Post by [deleted] »

maniacalhack0r wrote: post #35
reval wrote: post #19

Earlier quoted context omitted.

Why is that an anti-pattern? Databases have added `SKIP LOCKED` and `SELECT FOR UPDATE` to handle these use cases. What are the downsides?

as with everything, it depends on how you're processing the queue. eg we built a system at my last company to process 150 million objects / hour, and we modeled this using a postgres-backed queue with multiple processes pulling from the queue. we observed that, whenever there were a lot of locked rows (ie lots of work being done), Postgres would correctly SKIP these rows, but having to iterate over and skip that many…

[deleted]

Re: Kafka at the low end: how bad can it get?

#76

Post by relistan »

Joel_Mckay wrote: post #47
MuffinFlavored wrote: post #10

Earlier quoted context omitted.

NATS/WebSockets are good for 1 publisher -> many consumer (pubsub) RabbitMQ is good for 1 producer -> 1 consumer with ack/nack Right?

Actually, I used RabbitMQ static routes to feed per-cpu-core single thread bound consumers that restart their process every k transactions, or watchdog process timeout after w seconds. This prevents cross contamination of memory spaces, and slow fragmentation when the parsers get hammered hard. RabbitMQ/Erlang on OTP is probably one of the most solid solutions I've deployed over the years (low service cycle demands.)…

Agree. RabbitMQ is a Swiss Army knife that has a lot of available patterns, scales fairly well, and is very robust. If you don’t know what to choose, start with Rabbit. It will let you figure out which patterns you need and you probably won’t scale out of it. Pay someone to host it.

On the other hand, if you know what you need to do and it’s supported by it, NATS is IME the way to go (particularly JetStream).

Re: Kafka at the low end: how bad can it get?

#77

Post by InDubioProRubio »

atmosx wrote: post #68
NovemberWhiskey wrote: post #14

Kafka for small message volumes is one of those distinct resume-padding architectural vibes.

Oh no! Let’s be real: teams come to the infra team asking for a queue system. They give their requirements, and you—like a responsible engineer—suggest a more capable queue to handle their needs more efficiently. But no, they want Kafka. Kafka, Kafka, Kafka. Fine. You (meaning an entire team) set up Kafka clusters across three environments, define SLIs, enforce SLOs, make sure everything is production-grade. Then you…

Then all the guys who requested that stuff quit

Re: Kafka at the low end: how bad can it get?

#78

Post by InDubioProRubio »

araes wrote: post #34

Having never actually used this platform before, does anybody know why they named it Kafka, with all the horrible meanings? Per Wiktionary, Kafkaesque: [1] 1. "Marked by a senseless, disorienting, often menacing complexity." 2. "Marked by surreal distortion and often a sense of looming danger." 3. "In the manner of something written by Franz Kafka." (like the software language was written by Franz Kafka) Example: Met…

Because its a process: https://en.wikipedia.org/wiki/The_Trial

Re: Kafka at the low end: how bad can it get?

#79

Post by gnfargbl »

jszymborski wrote: post #3

What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.

Pulsar. Works extremely well as both a job queue and a data bus.

We have been using it in this application for half a decade now with no serious issues. I don't understand why it doesn't get more popular attention.

Re: Kafka at the low end: how bad can it get?

#80

Post by rizky05 »

InDubioProRubio wrote: post #77
atmosx wrote: post #68

Earlier quoted context omitted.

Oh no! Let’s be real: teams come to the infra team asking for a queue system. They give their requirements, and you—like a responsible engineer—suggest a more capable queue to handle their needs more efficiently. But no, they want Kafka. Kafka, Kafka, Kafka. Fine. You (meaning an entire team) set up Kafka clusters across three environments, define SLIs, enforce SLOs, make sure everything is production-grade. Then you…

Then all the guys who requested that stuff quit

[dead]
Post reply on HN