Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

31–40 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#31

Post by bencyoung »

zer00eyz wrote: post #3

> Should You Use Postgres? Most of the time - yes. You should always default to Postgres until the constraints prove you wrong. Kafka, GraphQL... These are the two technology's where my first question is always this: Does the person who championed/lead this project still work here? The answer is almost always "no, they got a new job after we launched". Resume Architecture is a real thing. Meanwhile the people left be…

Kafka is great tech, never sure why people have an issue with it. Would I use it all the time? No, but where it's useful, it's really useful, and opens up whole patterns that are hard to implement other ways

Re: Kafka is Fast – I'll use Postgres

#32

Post by sneilan1 »

candiddevmike wrote: post #18
sneilan1 wrote: post #9

I'm starting to like mongodb a lot more given the python library mongomock. I find it wonderful to create tests that run my queries against mongo in code before I deploy them. Yes, mongo has a lot of quirks and you have to know aws networking to set it up with your vpc so you don't get nailed with egress costs. And it's not the same query patterns and some queries are harder and you have maintain your own schemas. Bu…

Curious why you think the risk of edge cases from mocking is a worthwhile trade off vs the relatively low complexity of setting up a container to test against?

Because I can read the mongomock library and understand exactly what it's doing. And mongo's aggregation pipelines are easier to model than sql queries in code. Sure, it's possible to run into an edge case but for a lot of general queries for filtering & aggregation, it's just fine.

Re: Kafka is Fast – I'll use Postgres

#34

Post by Jupe »

jimbokun wrote: post #16

For me the killer feature of Kafka was the ability to set the offset independently for each consumer. In my company most of our topics need to be consumed by more than one application/team, so this feature is a must have. Also, the ability to move the offset backwards or forwards programmatically has been a life saver many times. Does Postgres support this functionality for their queues?

Isn't it just a matter of having each consumer use their own offset? I mean if the queue table is sequentially or time-indexed, the consumer just provides a smaller/earlier key to accomplish the offset? (Maybe I'm missing something here?)

Re: Kafka is Fast – I'll use Postgres

#35

Post by philipallstar »

sneilan1 wrote: post #9

I'm starting to like mongodb a lot more given the python library mongomock. I find it wonderful to create tests that run my queries against mongo in code before I deploy them. Yes, mongo has a lot of quirks and you have to know aws networking to set it up with your vpc so you don't get nailed with egress costs. And it's not the same query patterns and some queries are harder and you have maintain your own schemas. Bu…

You can also do this with sqlite, running an in-memory sqlite is lightning fast and I don't think there are any edge cases. Obviously doesn't work for everything, but when sqlite is possible, it's great!

Re: Kafka is Fast – I'll use Postgres

#37

Post by sneilan1 »

philipallstar wrote: post #35
sneilan1 wrote: post #9

I'm starting to like mongodb a lot more given the python library mongomock. I find it wonderful to create tests that run my queries against mongo in code before I deploy them. Yes, mongo has a lot of quirks and you have to know aws networking to set it up with your vpc so you don't get nailed with egress costs. And it's not the same query patterns and some queries are harder and you have maintain your own schemas. Bu…

You can also do this with sqlite, running an in-memory sqlite is lightning fast and I don't think there are any edge cases. Obviously doesn't work for everything, but when sqlite is possible, it's great!

True but if you wind up using parts of postgres that aren't supported by sqlite then it's harder to use sqlite. I agree however, if I was able to just use sqlite, I would do that instead. But I'm using a lot of postgres extensions & fields that don't have direct mappings to sqlite.

Otherwise SQLITE :)

Re: Kafka is Fast – I'll use Postgres

#38

Post by sitestable »

zer00eyz wrote: post #3

> Should You Use Postgres? Most of the time - yes. You should always default to Postgres until the constraints prove you wrong. Kafka, GraphQL... These are the two technology's where my first question is always this: Does the person who championed/lead this project still work here? The answer is almost always "no, they got a new job after we launched". Resume Architecture is a real thing. Meanwhile the people left be…

The best architecture decision is the one that's still maintainable when the person who championed it leaves. Always pretend the person who maintains a project after you knows where you live and all that.

Re: Kafka is Fast – I'll use Postgres

#39

Post by wosined »

ownagefool wrote: post #29

The camps are wrong. There's poles. 1. Is folks constantly adopting the new tech, whatever the motivation, and 2. I learned a thing and shall never learn anything else, ever. Of course nobody exists actually on either pole, but the closer you are to either, the less pragmatic you are likely to be.

I am the third pole: 3. Everything we have currently sucks and what is new will suck for some hitherto unknown reason.

Re: Kafka is Fast – I'll use Postgres

#40

Post by loire280 »

uberduper wrote: post #30

Has this person actually benchmarked kafka? The results they get with their 96 vcpu setup could be achieved with kafka on the 4 vcpu setup. Their results with PG are absurdly slow. If you don't need what kafka offers, don't use it. But don't pretend you're on to something with your custom 5k msg/s PG setup.

In fact, a properly-configured Kafka cluster on minimal hardware will saturate its network link before it hits CPU or disk bottlenecks.
Post reply on HN