Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

391–400 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#391

Post by bccdee »

otterley wrote: post #361
saberience wrote: post #347

Earlier quoted context omitted.

Think about this for a second. Kafka offsets are a thing, consumer groups are a thing. It's trivial to ensure that only one message is delivered to only one consumer if that's what you want. Consumer groups track their offset and then commit the offset, the message stays in Kafka but it won't be read again. This IMO is better behaviour than RabbitMQ since you can always re-read messages once they have been processed,…

> It's trivial to ensure that only one message is delivered to only one consumer Exactly-once delivery is one of the hardest distributed systems problems. If you’ve “trivially” solved it, please show us your solution.

> It's trivial to ensure that only one message is delivered to only one consumer if that's what you want. Consumer groups track their offset and then commit the offset, the message stays in Kafka but it won't be read again. This IMO is better behaviour than RabbitMQ

The trivial solution is to use Kafka. They're clearly saying that Kafka makes it trivial, not that it's trivial to solve from scratch.

Re: Kafka is Fast – I'll use Postgres

#392

Post by freedomben »

EdwardDiego wrote: post #283
freedomben wrote: post #149

Earlier quoted context omitted.

Indeed, I've seen this happen first hand where there was really only one guy who really "knew" Kafka, and it was too big of a job for just him. In that case it was fine until he left the company, and then it became a massive albatross and a major pain point. In another case, the eng team didn't really have anyone who really "knew" Kafka but used a managed service thinking it would be fine. It was until it wasn't, and…

I'm wondering why there wasn't any push for the Kafka guy to share his knowledge within his team, or to other teams?

Multiple factors (neither a good excuse, just reality):

* Lack of interest for other team members, which translated to doing what they thought was a sufficiently minimal amount of knowledge transfer

* An (unwise) attitude that "it's already set up and configured, and terraformed, so we can just acquire that knowledge if and when it's needed"

* Kafka guy left a lot faster than anybody really expected, not leaving much time and practically no documentation

* The rest of the team was already overwhelmed with other responsiblities and didn't have much bandwidth available

* Nobody wanted to be the person/people that ended up "owning" it, so there was a reverse incentive

Re: Kafka is Fast – I'll use Postgres

#393

Post by procaryote »

lumost wrote: post #255
procaryote wrote: post #214

Earlier quoted context omitted.

This is something to catch in hiring and performance evaluation. Hire people who don't build things to pad their own CVs, tell them to stop if you failed, fire them if that failed

Hiring irrational players, or forcing rational people to act outside of their own self-interest is not a winning strategy either. There is nothing wrong with building stuff, or career development. There is also nothing wrong with experimentation. You certainly would not want to incentivize the opposite behavior of never building anything unless it had 10 guarantors of revenue and technical soundness. If you need peop…

Some people do things for other reasons than maximising their monetary gain at every step; there are all kinds of motivations out there including liking solving problems

There's also a huge spectrum between "pick a job that's good for your career" and "at every step of the way I'll do whatever is best for me, the company and my coworkers be damned"

If you can't see that, just be open with it in the interview process

Re: Kafka is Fast – I'll use Postgres

#394

Post by procaryote »

59nadir wrote: post #304
procaryote wrote: post #214

Earlier quoted context omitted.

This is something to catch in hiring and performance evaluation. Hire people who don't build things to pad their own CVs, tell them to stop if you failed, fire them if that failed

This is a great way to get only people who basically can't build anything.

The people who use things they don't need to pad their CV haven't actually learnt the interesting parts. The only thing you can count on is that you'll get stuff added to your system to propel the person to their next role

Re: Kafka is Fast – I'll use Postgres

#396

Post by [deleted] »

bccdee wrote: post #391
otterley wrote: post #361

Earlier quoted context omitted.

> It's trivial to ensure that only one message is delivered to only one consumer Exactly-once delivery is one of the hardest distributed systems problems. If you’ve “trivially” solved it, please show us your solution.

> It's trivial to ensure that only one message is delivered to only one consumer if that's what you want. Consumer groups track their offset and then commit the offset, the message stays in Kafka but it won't be read again. This IMO is better behaviour than RabbitMQ The trivial solution is to use Kafka. They're clearly saying that Kafka makes it trivial, not that it's trivial to solve from scratch.

[deleted]

Re: Kafka is Fast – I'll use Postgres

#397

Post by otterley »

bccdee wrote: post #391
otterley wrote: post #361

Earlier quoted context omitted.

> It's trivial to ensure that only one message is delivered to only one consumer Exactly-once delivery is one of the hardest distributed systems problems. If you’ve “trivially” solved it, please show us your solution.

> It's trivial to ensure that only one message is delivered to only one consumer if that's what you want. Consumer groups track their offset and then commit the offset, the message stays in Kafka but it won't be read again. This IMO is better behaviour than RabbitMQ The trivial solution is to use Kafka. They're clearly saying that Kafka makes it trivial, not that it's trivial to solve from scratch.

What the parent poster described isn’t what makes Kafka’s “exactly once” semantics work. It’s the use of an idempotency token associated with each publication, which effectively turns “at-least-once” semantics into effectively “exactly once” via deduplication.

Re: Kafka is Fast – I'll use Postgres

#398

Post by jevyjevjevs »

gmokki wrote: post #386
saberience wrote: post #306

Earlier quoted context omitted.

Yes but the practical reality of it is it can be used exactly the same way as you would do a queue and you can make it work just as well as any MQ based system. I know this as I moved from a RabbitMQ system to Kafka for additionally scalability requirements and it worked perfectly. So sure "technically" it's not a queue, but in reality its used as a queue for 1000s of companies around the world for huge production wo…

Don't you always need a database after reading events from Kafka to deduplication? So the competing solutions are: PostgreSQL or Kafka+PostgreSQL Kafka does provide some extrs there, handling load spikes, more clients that PG can handle natively and resilience to some DB downtime. But is it worth the complexity, in most cases no.

Actually you can avoid having a separate DB! You can build a materialized view of the data using [KTables](https://developer.confluent.io/courses/kafka-streams/ktable/) or use [interactive queries](https://developer.confluent.io/courses/kafka-streams/interac...). The "table" is built up from a backing kafka topic so you don't need maintain another datastore if the data view you want is entirely derived from one or more Kafka topics.

Re: Kafka is Fast – I'll use Postgres

#399

Post by kragen »

freedomben wrote: post #392
EdwardDiego wrote: post #283

Earlier quoted context omitted.

I'm wondering why there wasn't any push for the Kafka guy to share his knowledge within his team, or to other teams?

Multiple factors (neither a good excuse, just reality): * Lack of interest for other team members, which translated to doing what they thought was a sufficiently minimal amount of knowledge transfer * An (unwise) attitude that "it's already set up and configured, and terraformed, so we can just acquire that knowledge if and when it's needed" * Kafka guy left a lot faster than anybody really expected, not leaving much…

Interesting, thanks!

Re: Kafka is Fast – I'll use Postgres

#400

Post by nonethewiser »

coldtea wrote: post #377
nonethewiser wrote: post #365

Earlier quoted context omitted.

I do not understand your position. I think it's a bit confused. >The Pareto principle is not some guarantee applicable to everything and anything Yes, obviously. The author doesn't say otherwise. There are obviously many ways of distributing things. >One can see how irrelevant its invocation is if we reverse: does Kafka also handle 80% of what Postgres does with 20% the effort? No >If not, what makes Postgres especia…

>I do not understand your position. Let me explain then... > Yes, obviously. The author doesn't say otherwise. Someone doesn't need to spell something out explicitly to imply it, or to fall to the kind of mistake I described. While the author might not say otherwise, they do invoke the Pareto principle out of context, as if it's some readily applicable theorem. >>If not, what makes Postgres especially the "Pareto 80%…

Where do they imply the Pareto principle is the only type of distribution?

The author's claim is that postgres handles 80% of the use cases with 20% of the effort. It does not follow, as you recognize, that therefor EVERYTHING handles 80% of the uses cases with 20% of the effort. Where does the author imply that it does? Based on his characterization of Kafka I think he's say it solves a minority of use cases with a lot more effort.

I mean isnt this the explicitly made point by the author? That Kafka's use-case/effort distribution is different (worse)?

Post reply on HN