Very interested to hear how people here overcome the limits of kafka for ordered events delivery in real world, and what those were.
put a timestamp in the message. use a conflict free replicated data type
Issues we've encountered while building a Kafka based data processing pipeline
51–60 of 96 posts
Re: Issues we've encountered while building a Kafka based data processing pipeline
#52Earlier quoted context omitted.
due to a bug Data retention time is Kafka config 101. Are you sure it was a bug?
Considering how half-assed Kafka is in general, that it needs all clients code changes when Kafka servers are upgraded. It is very likely that user hit Kafka bug.
Citation needed.
New server versions are protocol backwards compatible so I'm not sure what you're referring to.
Ofc, if you downgraded a server without changing the client, that may cause problems, but tbh that's hardly Kafka's fault.
Re: Issues we've encountered while building a Kafka based data processing pipeline
#53I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.
>My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.
Sorry thats just a clickbait-y statement. I love Postgres, try handling 100-500k rps of data coming in from various sources reading and writing to it. You are going to get bottlenecked on how many connections you can handle, you will end up throwing pgBouncers on top of it.
Eventually you will run out of disk, start throwing more in.
Then end up in VACCUUM hell all while having a single point of failure.
While I agree Kafka has its own issues, it is an amazing tool to a real scale problem.
Re: Issues we've encountered while building a Kafka based data processing pipeline
#54Earlier quoted context omitted.
using a sql db for push/pop semantic feels like using a hammer to squash a bug.. How would you model queues & partitions with ordering guarantees with pg ?
With transactions, and stored procedures if that helps ;). Redis also seems well suited to the use cases I've seen for Kafka. Kafka must have capabilities beyond those use cases, and I've sometimes wondered what they are.
[deleted]
Re: Issues we've encountered while building a Kafka based data processing pipeline
#55I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.
Can you kinda high level the setup/processes for making Postgres a replacement for Kafka? I've not attempted such a thing before, and wonder about things like expiration/autodeletion, etc. Does it need to be vacuumed often, and is that a problem?
Re: Issues we've encountered while building a Kafka based data processing pipeline
#56I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.
>My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres. Sorry thats just a clickbait-y statement. I love Postgres, try handling 100-500k rps of data coming in from various sources reading and writing to it. You are going to get bottlenecked on how many connections you can handle, you will end up throwing pgBouncers on top of it. Eventually you will run out of disk, star…
Yeah, once you do have to scale a relational database you're in for a world of pain. Band-aid after band-aid... I very much prefer to just start with Kafka already. At the very least you'll have a buffer to help you gain some time when the database struggles.
Re: Issues we've encountered while building a Kafka based data processing pipeline
#57I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.
>My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres. Sorry thats just a clickbait-y statement. I love Postgres, try handling 100-500k rps of data coming in from various sources reading and writing to it. You are going to get bottlenecked on how many connections you can handle, you will end up throwing pgBouncers on top of it. Eventually you will run out of disk, star…
I think anotherhue would agree that half a million write requests per second counts as a valid answer to "you can't do what you need with a beefy Postgres," but that is also a minority of situations.
Re: Issues we've encountered while building a Kafka based data processing pipeline
#58I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.
>My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres. Sorry thats just a clickbait-y statement. I love Postgres, try handling 100-500k rps of data coming in from various sources reading and writing to it. You are going to get bottlenecked on how many connections you can handle, you will end up throwing pgBouncers on top of it. Eventually you will run out of disk, star…
at least for me, every system has its place.
i love posgresql, but i would not use it to replace a rabbitmq instance -- one is an RDBMS, the other is a queue/event system.
"oh but psql can pretend to be kafka/rabbitmq!" -- sure, but then you need to add tooling to it, create libraries to handle it, and handle all the edge cases.
with rmq/kafka, there already a bunch of tools to handle the exact case of a queue/event system.
Re: Issues we've encountered while building a Kafka based data processing pipeline
#59I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.
Record an event log and reliably connect it to a variety of 3rd party sinks using off-the-shelf services
Re: Issues we've encountered while building a Kafka based data processing pipeline
#60Very interested to hear how people here overcome the limits of kafka for ordered events delivery in real world, and what those were.
put a timestamp in the message. use a conflict free replicated data type
If you need a guaranteed ordering, timestamps and distributed systems are not friends. See logical / vector clocks.