Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Issues we've encountered while building a Kafka based data processing pipeline

sixfold.medium.com

Post reply on HN
Print view View original

21–30 of 96 posts

Re: Issues we've encountered while building a Kafka based data processing pipeline

#21

Post by sumtechguy »

anotherhue wrote: post #2

I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.

I have done both patterns.

Kafka has its use case. Databases have theirs. You can make a DB do what kafka does. But you also add in the programming overhead of getting the DB semantics correct to make an event system. When I see people saying 'lets put the DB into kafka' I make the exact same argument. You will spend more time making kafka act like a database and getting the semantics right. Kafka is more of a data/event transportation system. A DB is an at rest data store that lets you manipulate the data. Use them to their strengths or get crushed by weird edge cases.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#22

Post by jgraettinger1 »

fafle wrote: post #8

The issue of running a transaction that spans multiple heterogeneous systems is usually solved with a 2 phase commit. The "jobs" abstraction from the article looks similar to the "coordinator" in 2PC. The article does not talk about how they achieve fault tolerance in case the "job" crashes inbetween the two transactions. Postgres supports the XA standard, which might help with this. Kafka does not support it.

I can't speak to their solution, but when solving an equivalent problem within Gazette, where you desire a distributed transaction that includes both a) published downstream messages, and b) state mutations in a DB, the solution is to 1) write downstream messages marked as pending a future ACK, and 2) encode the ACK you _intend_ to write into the checkpoint itself.

Commit the checkpoint alongside state mutations in a single store transaction. Only then do you publish ACKs to all of the downstream streams.

Of course, you can fail immediately after commit but before you get around to publishing all of those ACKS. So, on recovery, the first thing a task assignment does is publish (or re-publish) the ACKs encoded in the recovered checkpoint. This will either 1) provide a first notification that a commit occurred, or 2) be an effective no-op because the ACK was already observed, or 3) roll-back pending messages of a partial, failed transaction.

More details: https://gazette.readthedocs.io/en/latest/architecture-exactl...

Re: Issues we've encountered while building a Kafka based data processing pipeline

#23

Post by sumtechguy »

BFLpL0QNek wrote: post #10
bsaul wrote: post #3

Very interested to hear how people here overcome the limits of kafka for ordered events delivery in real world, and what those were.

It depends on what the events are, how they are structured. You get guaranteed ordering at the partition level. Items are partitioned by key so you also get guaranteed ordering for a key. If you have guaranteed ordering for a key you can’t get total ordering across all keys but you can get eventual consistency across the keys. Ultimately if you want ordering you have to design around being eventually consistent. I do…

For kafka the default is round robin in each partition. A hash key can let you direct the work to particular partitions. Each partition is guaranteed ordering. Also only one consumer in a consumer group can remove an item from a partition at a time. No two consumers in a consumer group will get the same message.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#24

Post by HelloNurse »

The only time I used Kafka, it was involuntarily (included for the sake of fashion in some complicated IBM product, where it hid among WebSphere, DB2 and other bigger elephants) and it ran my server out of disk space because due to a bug ridiculously massive temporary files weren't erased. Needless to say, I wasn't impressed: just one more hazard to worry about.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#25

Post by jgraettinger1 »

bsaul wrote: post #3

Very interested to hear how people here overcome the limits of kafka for ordered events delivery in real world, and what those were.

Not for Kafka, but we are building Flow [1] to offer deterministic ordering, even across multiple logical and physical partitions, and regardless of whether you're back-filling over history or processing in real-time.

This ends up being required for one of our architectural goals, which is fully repeatable transformations: You must be able to model a transactional decision as a Flow derivation (like "does account X have funds to transfer Y to Z ?", and if you create a _copy_ of that derivation months later, get the exact same result.

Under the hood (and simplifying a bit) Flow always does a streaming shuffled read to map events from partitions to task shards, and each shard maintains a min-heap to process events in their ~wall-time order.

This also avoids the common "Tyranny of Partitioning", where your upstream partitioning parallelism N also locks you into that same task shard parallelism -- a big problem if tasks manage a lot of state. With a read-time shuffle, you can scale them independently.

[1]: https://github.com/estuary/flow

Re: Issues we've encountered while building a Kafka based data processing pipeline

#26

Post by ThinkBeat »

anotherhue wrote: post #2

I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.

This is exactly how I feel about it.

A while back I was on team building a non-critical, low volume application. It just involves people sending a message basically. (there is more too it).

The consultants said he had to use Kafka because the messages could come in really fast.

I said we should stick with Postgres.

No, they said, we really need Kakfa to be able to handle this.

Then I went and spun up Postgres on my work laptop (nothing special), and got a loaner to act as a client. I simulated about 300% more traffic than we had any chance of getting. It worked fine. (did tax my poor work laptop).

No, we could not risk it, when we use Kafka we are safe.

Took it to management, Kafka won since Buzzword.

Now of course we have to write a process to feed the data into Postgres. After all its what everything else depends on

Re: Issues we've encountered while building a Kafka based data processing pipeline

#27

Post by gilbetron »

anotherhue wrote: post #2

I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.

What velocity have you achieved with postgres, in terms of # messages/sec where messages could range between 1KB-100KB in size?

Re: Issues we've encountered while building a Kafka based data processing pipeline

#28

Post by sparsely »

ThinkBeat wrote: post #26
anotherhue wrote: post #2

I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.

This is exactly how I feel about it. A while back I was on team building a non-critical, low volume application. It just involves people sending a message basically. (there is more too it). The consultants said he had to use Kafka because the messages could come in really fast. I said we should stick with Postgres. No, they said, we really need Kakfa to be able to handle this. Then I went and spun up Postgres on my w…

There are various Kafka to Postgres adaptors. Of course, now you're running 3 bits of software, with 3 bottlenecks, instead of just 1.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#29

Post by dtech »

ThinkBeat wrote: post #26
anotherhue wrote: post #2

I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.

This is exactly how I feel about it. A while back I was on team building a non-critical, low volume application. It just involves people sending a message basically. (there is more too it). The consultants said he had to use Kafka because the messages could come in really fast. I said we should stick with Postgres. No, they said, we really need Kakfa to be able to handle this. Then I went and spun up Postgres on my w…

Were you advocating an endpoint + DB or different apps directly writing into a shared DB? The latter is not really a good idea for numerous reasons. Kafka a - potentially overkill - replacement for REST or whatever, not for your DB.

Re: Issues we've encountered while building a Kafka based data processing pipeline

#30

Post by lima »

gilbetron wrote: post #27
anotherhue wrote: post #2

I ran a few dozen kafka clusters at MegaCorp in a previous life. My answer to anyone who asks for kafka: Show me that you can't do what you need with a beefy Postgres.

What velocity have you achieved with postgres, in terms of # messages/sec where messages could range between 1KB-100KB in size?

Yup. That's what Kafka excels at and where it scales to throughput way beyond Postgres.

But there are many many projects where Kafka is used for low value event sourcing stuff where a SQL DB could be easier.

Post reply on HN