Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Issues we've encountered while building a Kafka based data processing pipeline

sixfold.medium.com

Post reply on HN
Print view View original

71–80 of 96 posts

Re: Issues we've encountered while building a Kafka based data processing pipeline

#71

Post by anotherhue »

BiteCode_dev wrote: post #40
sumtechguy wrote: post #21

Earlier quoted context omitted.

I have done both patterns. Kafka has its use case. Databases have theirs. You can make a DB do what kafka does. But you also add in the programming overhead of getting the DB semantics correct to make an event system. When I see people saying 'lets put the DB into kafka' I make the exact same argument. You will spend more time making kafka act like a database and getting the semantics right. Kafka is more of a data/e…

Well, if you want events, you have lighter alternative. Redis pub/sub, crossbar... Even rabbitMQ is lighter. If you need queuing, you have libs like celery. You don't need to go full kafka

Agreed. Redis pub/sub is underrated.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#72

Post by dwohnitmok »

IggleSniggle wrote: post #61
dwohnitmok wrote: post #57

Earlier quoted context omitted.

I think anotherhue would agree that half a million write requests per second counts as a valid answer to "you can't do what you need with a beefy Postgres," but that is also a minority of situations.

It's just hard to know what people mean when they say "most people don't need to do this." I was sitting wondering about a similar scale (200-1000 rps), where I've had issues with scaling rabbitmq, and have been thinking about whether kafka might help. Without context provided, you might think: "oh, here's somebody with kafka and postgres experience, saying that postgres has some other super powers I hadn't learned a…

Note superyesh was talking about 100 to 500 thousand requests per second. Your overall question stands, but the scale superyesh was talking about is very different and I am quite confident superyesh's scale is definitely in the minority.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#73

Post by Cederfjard »

IggleSniggle wrote: post #61
dwohnitmok wrote: post #57

Earlier quoted context omitted.

I think anotherhue would agree that half a million write requests per second counts as a valid answer to "you can't do what you need with a beefy Postgres," but that is also a minority of situations.

It's just hard to know what people mean when they say "most people don't need to do this." I was sitting wondering about a similar scale (200-1000 rps), where I've had issues with scaling rabbitmq, and have been thinking about whether kafka might help. Without context provided, you might think: "oh, here's somebody with kafka and postgres experience, saying that postgres has some other super powers I hadn't learned a…

I’m not sure if you’re omitting the k in your numbers, or missed it in the other comment? Do you mean 100-500 and 200-1000, or 100 000-500 000 and 200 000-1 000 000?

Re: Issues we've encountered while building a Kafka based data processing pipeline

#74

Post by anotherhue »

ceencee wrote: post #36
anotherhue wrote: post #2

I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.

I see posts like this a lot, and it makes me wonder what the heck you were using Kafka for that Postgres could handle, yet you had dozens of clusters? I question if you actually ever used Kafka or just operated it? Sure anyone can follow the "build a queue on a database pattern" but it falls over at the throughputs that justify Kafka. If you have a bunch of trivial 10tps workloads, of course a distributed system is o…

A major ecommerce site. We had hundreds of thousands of messages/s but for most use cases YAGNI.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#75

Post by Cederfjard »

ceencee wrote: post #36
anotherhue wrote: post #2

I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.

I see posts like this a lot, and it makes me wonder what the heck you were using Kafka for that Postgres could handle, yet you had dozens of clusters? I question if you actually ever used Kafka or just operated it? Sure anyone can follow the "build a queue on a database pattern" but it falls over at the throughputs that justify Kafka. If you have a bunch of trivial 10tps workloads, of course a distributed system is o…

They didn’t say that the Kafka clusters they personally ran could have been handled with Postgres instead.

They first gave their credentials by mentioning their experience.

Then they basically said ”given what I know about Kafka, with my experience, I require other people who ask for it to show me that they really need it before I accommodate them - often a beefy Postgres is enough”.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#76

Post by EdwardDiego »

tiew9Vii wrote: post #66
sumtechguy wrote: post #23

Earlier quoted context omitted.

For kafka the default is round robin in each partition. A hash key can let you direct the work to particular partitions. Each partition is guaranteed ordering. Also only one consumer in a consumer group can remove an item from a partition at a time. No two consumers in a consumer group will get the same message.

It's round robin if no key specified otherwise it uses murmur2 hash of the key so the partition for a key is always deterministic. Just checking the docs it appears the round robin is no longer true after Confluent Platform 5.4. After 5.4 it looks like if no key specified the partition is assigned based on the batch being processed. > If the key is provided, the partitioner will hash the key with murmur2 algorithm an…

Yep, IIRC around Kafka 2.6? the default partitioner changed from RoundRobin to preferring existing open record batches.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#77

Post by [deleted] »

geodel wrote: post #49
kitd wrote: post #44

Earlier quoted context omitted.

due to a bug Data retention time is Kafka config 101. Are you sure it was a bug?

Considering how half-assed Kafka is in general, that it needs all clients code changes when Kafka servers are upgraded. It is very likely that user hit Kafka bug.

[deleted]

Re: Issues we've encountered while building a Kafka based data processing pipeline

#78

Post by EdwardDiego »

geodel wrote: post #49
kitd wrote: post #44

Earlier quoted context omitted.

due to a bug Data retention time is Kafka config 101. Are you sure it was a bug?

Considering how half-assed Kafka is in general, that it needs all clients code changes when Kafka servers are upgraded. It is very likely that user hit Kafka bug.

> it needs all clients code changes when Kafka servers are upgraded

It absolutely doesn't. Message formats predating Kafka 0.11 are only just being deprecated as of Kafka 3.0, and won't be dropped until Kafka 4.0.

Now, if you want to use new shiny features (like cooperative sticky assignors to minimise consumer group stop the world rebalance pauses), then yes, you might need to upgrade clients.

But otherwise, you can still happily use 0.8 clients with your upgraded brokers.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A...

Re: Issues we've encountered while building a Kafka based data processing pipeline

#79

Post by EdwardDiego »

HelloNurse wrote: post #24

The only time I used Kafka, it was involuntarily (included for the sake of fashion in some complicated IBM product, where it hid among WebSphere, DB2 and other bigger elephants) and it ran my server out of disk space because due to a bug ridiculously massive temporary files weren't erased. Needless to say, I wasn't impressed: just one more hazard to worry about.

What was the bug, out of curiosity?

Re: Issues we've encountered while building a Kafka based data processing pipeline

#80

Post by rajin444 »

staticassertion wrote: post #68

(1) seems best solved by having a 'on heavy task, publish to a secondary topic'. This is good if you have flaky messages that need to be retried in the background, without blocking all of your 'good' messages. (2) this problem should be avoided in general by just having idempotent services. Just as a hard restriction, forever, build services to be idempotent. It should be the exception to have a non-idempotent servic…

This seems like a much simpler solution. The "heavy task queue" would process tasks that could simply be retried until they're done.

Maybe I'm misunderstanding the article, but having "Job tasks" both insert another Job to run as well as updating DB state, and then having the executor pick up the previously inserted Job (whos only purpose is to send a kafka message) seems overly complex. I'm having trouble seeing why this is needed.

Post reply on HN