Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

361–370 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#361

Post by otterley »

saberience wrote: post #347
dxxvi wrote: post #337

Earlier quoted context omitted.

Is it true that a message from a queue will disappear after it is consumed successfully? If yes, at this moment, how do you make kafka topics work as queues?

Think about this for a second. Kafka offsets are a thing, consumer groups are a thing. It's trivial to ensure that only one message is delivered to only one consumer if that's what you want. Consumer groups track their offset and then commit the offset, the message stays in Kafka but it won't be read again. This IMO is better behaviour than RabbitMQ since you can always re-read messages once they have been processed,…

> It's trivial to ensure that only one message is delivered to only one consumer

Exactly-once delivery is one of the hardest distributed systems problems. If you’ve “trivially” solved it, please show us your solution.

Re: Kafka is Fast – I'll use Postgres

#362

Post by otterley »

saberience wrote: post #345
guhcampos wrote: post #338

Earlier quoted context omitted.

It is a *very bad* replacement for an MQ system, for the simple reason you can't quickly and effortlessly scale int/out consumers.

Why can't you? In my experience, scaling Kafka was far easier than scaling our RabbitMQ cluster. We started running into issues when our RabbitMQ cluster hit 25k TPS, our Kafka cluster of equivalent resources didn't break a sweat at 500k TPS.

Scaling consumers, not throughput. And it’s both directions (in and out, not just out).

Re: Kafka is Fast – I'll use Postgres

#363

Post by LgWoodenBadger »

rileymichael wrote: post #357
saberience wrote: post #306

Earlier quoted context omitted.

Yes but the practical reality of it is it can be used exactly the same way as you would do a queue and you can make it work just as well as any MQ based system. I know this as I moved from a RabbitMQ system to Kafka for additionally scalability requirements and it worked perfectly. So sure "technically" it's not a queue, but in reality its used as a queue for 1000s of companies around the world for huge production wo…

> you can make it work just as well as any MQ based system you really can't. getting per-message acks, dynamically scaling competing consumers without having to repartition while retaining ordering, etc. requires a ton of hacks like client side tracking / building your own storage on top of offset metadata / etc.. and you still won't have all of the features actual message queues provide. to make it worse, there is v…

I don't think "while retaining ordering" is something you want to include here, since you can't guarantee processing order for any MQ system without serializing the consumption down to a single consumer.

Re: Kafka is Fast – I'll use Postgres

#364

Post by williamdclt »

throwwgisgreat wrote: post #353
ARandomerDude wrote: post #339

I'm solidly in camp 2, the "common sense" camp that doesn't care about buzzwords. That said, I don't consider running Kafka to be a headache. I work at a mid-sized company, processing billions of Kafka events per day and it's never been a problem, even locally when I'm processing hundreds of events per day. You set it up, forget about it, and it scales endlessly. You don't have to rewrite anything and it provides a n…

> processing billions of Kafka events per day Except that the burden is on all clients to coordinate to avoid processing an event more than once since Kakfa is a brainless invention just dumping data forever into a serial log.

I'm not sure what you're talking about.

Do you mean different consumers within the same consumer group? There's no technology out there that will guarantee exactly-once delivery, it's simply impossible in a world where networks aren't magically 100% reliable. SQS, RedPanda, RabbitMQ, NATS... you call it, your client will always need idempotency.

Re: Kafka is Fast – I'll use Postgres

#365

Post by nonethewiser »

coldtea wrote: post #316

> The claim is that it handles 80%+ of their use cases with 20% of the development effort. (Pareto Principle) The Pareto principle is not some guarantee applicable to everything and anything saying that any X will handle 80% of some other thing's use cases with 20% the effort. One can see how irrelevant its invocation is if we reverse: does Kafka also handle 80% of what Postgres does with 20% the effort? If not, what…

I do not understand your position. I think it's a bit confused.

>The Pareto principle is not some guarantee applicable to everything and anything

Yes, obviously. The author doesn't say otherwise. There are obviously many ways of distributing things.

>One can see how irrelevant its invocation is if we reverse: does Kafka also handle 80% of what Postgres does with 20% the effort?

No

>If not, what makes Postgres especially the "Pareto 80%" one in this comparison?

Because its simpler.

What implies everything can handle 80% of use cases with 20% of effort? It's like saying:

If normal distributions are real, and human height is normally distributed, then why isnt personal wealth? They are just different distributions.

Re: Kafka is Fast – I'll use Postgres

#366

Post by BinaryIgor »

That's golden:

"2. The other camp chases common sense

This camp is far more pragmatic. They strip away unnecessary complexity and steer clear of overengineered solutions. They reason from first principles before making technology choices. They resist marketing hype and approach vendor claims with healthy skepticism."

We should definitely apply Occam's razor as the industry far more often; simple tech stacks are better to manage and especially master (which you must do, once it's no longer a toy app). Introduce a new component into your system only if it provides functionality you cannot get with reasonable effort, using what you already have.

Re: Kafka is Fast – I'll use Postgres

#367

Post by BinaryIgor »

munchbunny wrote: post #119

My general opinion, off the cuff, from having worked at both small (hundreds of events per hour) and large (trillions of events per hour) scales for these sorts of problems: 1. Do you really need a queue? (Alternative: periodic polling of a DB) 2. What's your event volume and can it fit on one node for the foreseeable future, or even serverless compute (if not too expensive)? (Alternative: lightweight single-process…

100%; often batch integration - pulling data from REST endpoint/endpoints in the background, with reasonable frequency like every 5 -15 minutes - is good enough ;)

Re: Kafka is Fast – I'll use Postgres

#368

Post by BinaryIgor »

agentultra wrote: post #6

You have to be careful with the approach of using Postgres for everything. The way it locks tables and rows and the serialization levels it guarantees are not immediately obvious to a lot of folks and can become a serious bottle-neck for performance-sensitive workloads. I've been a happy Postgres user for several decades. Postgres can do a lot! But like anything, don't rely on maxims to do your engineering for you.

True, but you have to have a really intensive workload to hit its limits; something in the order of tens of thousands writes per second; and even then, you can shard to a few instances. So yes, there is a limit - but in practice, not for most systems

Re: Kafka is Fast – I'll use Postgres

#369

Post by jdswain »

saberience wrote: post #345
guhcampos wrote: post #338

Earlier quoted context omitted.

It is a *very bad* replacement for an MQ system, for the simple reason you can't quickly and effortlessly scale int/out consumers.

Why can't you? In my experience, scaling Kafka was far easier than scaling our RabbitMQ cluster. We started running into issues when our RabbitMQ cluster hit 25k TPS, our Kafka cluster of equivalent resources didn't break a sweat at 500k TPS.

What sort of systems do you work on to require this kind of traffic volume? I've worked on one project that I'd consider relatively high volume (UK Post Office Horizon Online) and we were only targeting 500 TPS.

Re: Kafka is Fast – I'll use Postgres

#370

Post by rileymichael »

LgWoodenBadger wrote: post #363
rileymichael wrote: post #357

Earlier quoted context omitted.

> you can make it work just as well as any MQ based system you really can't. getting per-message acks, dynamically scaling competing consumers without having to repartition while retaining ordering, etc. requires a ton of hacks like client side tracking / building your own storage on top of offset metadata / etc.. and you still won't have all of the features actual message queues provide. to make it worse, there is v…

I don't think "while retaining ordering" is something you want to include here, since you can't guarantee processing order for any MQ system without serializing the consumption down to a single consumer.

absolute order, you're correct. but several systems support demultiplexing an interleaved stream by some key so that order is retained for all messages in each key space (pulsar key_shared subscriptions, azure service bus sessions, etc). you're still serializing consumption / have an exclusive consumer for each key, but with global parallelism.

here's the equivalent in parallel-consumer: https://github.com/confluentinc/parallel-consumer?tab=readme...

Post reply on HN