Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

351–360 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#351

Post by enether »

augusto-moura wrote: post #344
coldtea wrote: post #316

> The claim is that it handles 80%+ of their use cases with 20% of the development effort. (Pareto Principle) The Pareto principle is not some guarantee applicable to everything and anything saying that any X will handle 80% of some other thing's use cases with 20% the effort. One can see how irrelevant its invocation is if we reverse: does Kafka also handle 80% of what Postgres does with 20% the effort? If not, what…

I believe the Pareto principle is used more of a figure of speech nowadays, it doesn't mean the 80 to 20 ratio literally. When I hear someone invoking the principle I tend to think of it as: this should solve the problem enough for our case and with less (usually much less) work than a complete solution.

yeah, this was my intention. I have no real way of knowing if it truly covers 80% of use cases, or 70%, or 40%, anyway.

Re: Kafka is Fast – I'll use Postgres

#352

Post by enether »

ARandomerDude wrote: post #339

I'm solidly in camp 2, the "common sense" camp that doesn't care about buzzwords. That said, I don't consider running Kafka to be a headache. I work at a mid-sized company, processing billions of Kafka events per day and it's never been a problem, even locally when I'm processing hundreds of events per day. You set it up, forget about it, and it scales endlessly. You don't have to rewrite anything and it provides a n…

I also strongly believe it's not a headache.

Vendors frequently push that narrative so they can sell their own managed (or proprietary) solution on it. With a decent AI model (e.g ChatGPT Pro), it's easier than ever to figure out best practices and conventions.

That being said, my point is more about the organizational overhead. Deploying Kafka still means you need to learn how it works, why it's good, its configs, API, how to debug it, set up obesrvability, yada yada.

Re: Kafka is Fast – I'll use Postgres

#353

Post by throwwgisgreat »

ARandomerDude wrote: post #339

I'm solidly in camp 2, the "common sense" camp that doesn't care about buzzwords. That said, I don't consider running Kafka to be a headache. I work at a mid-sized company, processing billions of Kafka events per day and it's never been a problem, even locally when I'm processing hundreds of events per day. You set it up, forget about it, and it scales endlessly. You don't have to rewrite anything and it provides a n…

> processing billions of Kafka events per day

Except that the burden is on all clients to coordinate to avoid processing an event more than once since Kakfa is a brainless invention just dumping data forever into a serial log.

Re: Kafka is Fast – I'll use Postgres

#354

Post by throwwgisgreat »

abtinf wrote: post #247
skunkworker wrote: post #238

Earlier quoted context omitted.

I wish postgres would add a durable queue like data structure. But trying to make a durable queue that can scale beyond what a simple redis instance can do starts to run into problems quickly. Also, LISTEN/NOTIFY do not scale, and they introduce locks in areas you aren't expecting - https://news.ycombinator.com/item?id=44490510

SKIP LOCKED doesn't work for your use case?

It would probably work fine, it would also put the jobs at risk of people who managed to convince their enterprises that a dumb but fast server (Kafka) was actually a good idea.

Re: Kafka is Fast – I'll use Postgres

#355

Post by throwwgisgreat »

saberience wrote: post #347
dxxvi wrote: post #337

Earlier quoted context omitted.

Is it true that a message from a queue will disappear after it is consumed successfully? If yes, at this moment, how do you make kafka topics work as queues?

Think about this for a second. Kafka offsets are a thing, consumer groups are a thing. It's trivial to ensure that only one message is delivered to only one consumer if that's what you want. Consumer groups track their offset and then commit the offset, the message stays in Kafka but it won't be read again. This IMO is better behaviour than RabbitMQ since you can always re-read messages once they have been processed,…

> better behaviour than RabbitMQ since you can always re-read messages once they have been processed

I can imagine, a 1 Billion dollar transaction accidentally gets processed by ten thousand client nodes due to a client app synchronization bug, company rethinks its dumb data dumper server strategy...news at 11.

Re: Kafka is Fast – I'll use Postgres

#356

Post by riku_iki »

AtlasBarfed wrote: post #235
javier2 wrote: post #211

Earlier quoted context omitted.

Postgres doesnt scale into oblivion, but it can take some serious chunks of data once you start batching and making sure a every operation only touches single row with no transactions needed.

And then you are 99% of the way to Cassandra. Of course the other 99% is the remaining 1%.

cassandra doesn't have ACID, so you will start dealing with tons of other problems.

Re: Kafka is Fast – I'll use Postgres

#357

Post by rileymichael »

saberience wrote: post #306
EdwardDiego wrote: post #271

Earlier quoted context omitted.

Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement for an MQ, and it's such the wrong way to think about it.

Yes but the practical reality of it is it can be used exactly the same way as you would do a queue and you can make it work just as well as any MQ based system. I know this as I moved from a RabbitMQ system to Kafka for additionally scalability requirements and it worked perfectly. So sure "technically" it's not a queue, but in reality its used as a queue for 1000s of companies around the world for huge production wo…

> you can make it work just as well as any MQ based system

you really can't. getting per-message acks, dynamically scaling competing consumers without having to repartition while retaining ordering, etc. requires a ton of hacks like client side tracking / building your own storage on top of offset metadata / etc.. and you still won't have all of the features actual message queues provide.

to make it worse, there is very little public work/discussion on this so you'll be on your own to figure out all of the quirks. the only notable example is https://github.com/confluentinc/parallel-consumer which is effectively abandoned

Re: Kafka is Fast – I'll use Postgres

#358

Post by LgWoodenBadger »

dxxvi wrote: post #337
saberience wrote: post #306

Earlier quoted context omitted.

Yes but the practical reality of it is it can be used exactly the same way as you would do a queue and you can make it work just as well as any MQ based system. I know this as I moved from a RabbitMQ system to Kafka for additionally scalability requirements and it worked perfectly. So sure "technically" it's not a queue, but in reality its used as a queue for 1000s of companies around the world for huge production wo…

Is it true that a message from a queue will disappear after it is consumed successfully? If yes, at this moment, how do you make kafka topics work as queues?

It "disappears" in the sense that the Consumer-Group that read/committed that message (event) will never see it again. It doesn't "disappear" in the sense that a new Consumer-Group can be started in a way that will get that message, or you can reset your Consumer-Group's offset to re-consume it.

Re: Kafka is Fast – I'll use Postgres

#359

Post by LgWoodenBadger »

EdwardDiego wrote: post #286
codeflo wrote: post #280

Earlier quoted context omitted.

Do you mean this in the sense that listeners don't remove messages, as one would expect from a queue data structure?

Exactly. There's no concept in Kafka (yet...) of "acking" or DLQs, Kafka is very good at what it does by being deliberately stupid, it knows nothing about your messages or who has consumed them and who hasn't. That was all deliberately pushed onto consumers to manage to achieve scale.

What do you mean "no concept...of asking?" During consumption, you must either auto-commit offsets, or manually commit them. If you don't, you'll get the same events over and over again.

Re: Kafka is Fast – I'll use Postgres

#360

Post by natmaka »

IMHO the main difference between PostgreSQL and any 'competitor' is that in most cases a software developer will quickly find not only how to use it quite properly for his use case but also why some way he adopted isn't right and triggers some non-negligible problem.

There are many reasons for this: most software developers have more than a vague idea about its underlying concepts, most error messages are clear, the documentation is superb, there are many ways to tap into the vast knowledge of a huge and growing community...

Post reply on HN