Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

331–340 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#331

Post by mike_hearn »

sneilan1 wrote: post #13
agentultra wrote: post #6

You have to be careful with the approach of using Postgres for everything. The way it locks tables and rows and the serialization levels it guarantees are not immediately obvious to a lot of folks and can become a serious bottle-neck for performance-sensitive workloads. I've been a happy Postgres user for several decades. Postgres can do a lot! But like anything, don't rely on maxims to do your engineering for you.

Yes, performance can be a big issue with postgres. And vertical scaling can really put a damper on things when you have a major traffic hit. Using it for kafka is misunderstanding the one of the great uses of kafka which is to help deal with traffic bursts. All of a sudden your postgres server is overwhelmed and the kafka server would be fine.

It's worth noting that Oracle has solved this problem. It has horizontal multi-master scalability (not sharded) and a queue subsystem called TxEQ which scales like Kafka does, but it's also got the features of a normal MQ broker. You can dequeue a message into a transaction, update tables in that same transaction, then commit to remove the message from the queue permanently. You can dequeue by predicate, delay messages, use producer/consumer patterns etc. It's quite flexible. The queues can be accessed via SQL stored procs, or client driver APIs, or it implements a Kafka compatible API now too I think.

If you rent a cloud DB then it can scale elastically which can make this cheaper than Postgres, believe it or not. Cloud databases are sold at the price the market will bear not the cost of inputs+margin, so you can end up paying for Postgres as much as you would for an Oracle DB whilst getting far fewer features and less scalability.

Source: recently joined the DB team at Oracle, was surprised to learn how much it can do.

Re: Kafka is Fast – I'll use Postgres

#332

Post by Nifty3929 »

I do agree that too often folks are looking for the cool new widget and looking to apply it to every problem, with fancy new "modernized" architectures and such. And Postgres is great for so much.

But I think an important point to those in camp 2 (the good guys in TFA's narrative) is to use tools for problems they were designed to solve. Postgres was not designed to be a pub-sub tool. Kafka was. Don't try to build your own pub-sub solution on top of Postgres, just use one of the products that was built for that job.

Another distressing trend I see is for every product to try to be everything to everyone. I do not need that. I just need your product to do it's one thing very well, and then I will use a different product for a different thing I need.

Re: Kafka is Fast – I'll use Postgres

#333

Post by ses1984 »

MrDarcy wrote: post #329
coldtea wrote: post #316

> The claim is that it handles 80%+ of their use cases with 20% of the development effort. (Pareto Principle) The Pareto principle is not some guarantee applicable to everything and anything saying that any X will handle 80% of some other thing's use cases with 20% the effort. One can see how irrelevant its invocation is if we reverse: does Kafka also handle 80% of what Postgres does with 20% the effort? If not, what…

Is the mapping of use cases to software functionality not a power law distribution? Meaning there are a few use cases that have a disproportionate affect on the desired outcome if provided by the software?

You might be right, but does anyone have data to support that hypothesis?

Re: Kafka is Fast – I'll use Postgres

#334

Post by JohnBooty »

jsolson wrote: post #265
munchbunny wrote: post #119

My general opinion, off the cuff, from having worked at both small (hundreds of events per hour) and large (trillions of events per hour) scales for these sorts of problems: 1. Do you really need a queue? (Alternative: periodic polling of a DB) 2. What's your event volume and can it fit on one node for the foreseeable future, or even serverless compute (if not too expensive)? (Alternative: lightweight single-process…

I agree with nearly everything except your point (1). Periodic polling is awkward on both sides: you add arbitrary latency _and_ increase database load proportional to the number of interested clients. Events, and ideally coalesced events, serve the same purpose as interrupts in a uniprocess (versus distributed) system, even if you don't want a proper queue. This at least lets you know _when_ to poll and lets you set…

From a database load perspective, Postgres can get you pretty far. The reads triggered by each poll should be trivial index-only scans served right out of RAM. Even a modest Postgres instance should be able to handle thousands per second.

The limiting factor for most workloads will probably be the number of connections, and the read/write mix. When you get into hundreds or thousands of pollers and writing many things to the queue per second Postgres is going to lose its luster for sure.

But in my experience with small/medium companies, a lot of workloads fit very very comfortably into what Postgres can handle easily.

Re: Kafka is Fast – I'll use Postgres

#335

Post by xmcqdpt2 »

notepad0x90 wrote: post #172
oulipo2 wrote: post #124

Earlier quoted context omitted.

I want to rewrite some of my setup, we're doing IoT, and I was planning on MQTT -> Redpanda (for message logs and replay, etc) -> Postgres/Timescaledb (for data) + S3 (for archive) (and possibly Flink/RisingWave/Arroyo somewhere in order to do some alerting/incrementally updated materialized views/ etc) this seems "simple enough" (but I don't have any experience with Redpanda) but is indeed one more moving part compa…

Another good item to consider: n) Do you really need S3? is it cheaper than NFS storage on a compute node with a large disk? There are many cases where S3 is absolutely cheaper though.

In my experience NFS is always the wrong thing to use.

Your application think it's a normal disk but it isn't, so you get no timeouts, no specific errors for network issues and extremely expensive calls camouflage as quick FS ops (was any file modified in this folder ? I'll just loop over them using my standard library nice FS utilities). And you don't get atomic ops outside of mv, invalidation and caching are complicated and your developers probably don't know the semantics of FS operations, which are much more complex and less well documented than eg Redis blob storage.

And then when you finally rip out NFS, you have thousands of lines of app and test code that assumes your blobs are on a disk in subtle ways.

Re: Kafka is Fast – I'll use Postgres

#336

Post by notpaulgraham »

sgarland wrote: post #322
animuchan wrote: post #317

Earlier quoted context omitted.

LLM solves this by meeting devs in the middle: the vibe coded DB schema, coupled with agentically-made application code, makes even 20,000 records a "huge scale".

This is so accurate. I’ve looked in wonder at how someone is maxing out an r6i.32xlarge MySQL DB, when I have ran 4x the workload on an r6i.12xlarge. Schema design and query design will make or break your app’s ability to scale without skyrocketing the bill, it’s as simple as that.

any blogs/books you'd recommend on schema & query design? it honestly surprises me that these coding-focused models can't look at a schema; look at how data is being queried; reason about the use case for the data; and help prioritize solving for the most likely bottlenecks to scaling the underlying data services.

Re: Kafka is Fast – I'll use Postgres

#337

Post by dxxvi »

saberience wrote: post #306
EdwardDiego wrote: post #271

Earlier quoted context omitted.

Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement for an MQ, and it's such the wrong way to think about it.

Yes but the practical reality of it is it can be used exactly the same way as you would do a queue and you can make it work just as well as any MQ based system. I know this as I moved from a RabbitMQ system to Kafka for additionally scalability requirements and it worked perfectly. So sure "technically" it's not a queue, but in reality its used as a queue for 1000s of companies around the world for huge production wo…

Is it true that a message from a queue will disappear after it is consumed successfully? If yes, at this moment, how do you make kafka topics work as queues?

Re: Kafka is Fast – I'll use Postgres

#338

Post by guhcampos »

saberience wrote: post #306
EdwardDiego wrote: post #271

Earlier quoted context omitted.

Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement for an MQ, and it's such the wrong way to think about it.

Yes but the practical reality of it is it can be used exactly the same way as you would do a queue and you can make it work just as well as any MQ based system. I know this as I moved from a RabbitMQ system to Kafka for additionally scalability requirements and it worked perfectly. So sure "technically" it's not a queue, but in reality its used as a queue for 1000s of companies around the world for huge production wo…

It is a *very bad* replacement for an MQ system, for the simple reason you can't quickly and effortlessly scale int/out consumers.

Re: Kafka is Fast – I'll use Postgres

#339

Post by ARandomerDude »

I'm solidly in camp 2, the "common sense" camp that doesn't care about buzzwords.

That said, I don't consider running Kafka to be a headache. I work at a mid-sized company, processing billions of Kafka events per day and it's never been a problem, even locally when I'm processing hundreds of events per day.

You set it up, forget about it, and it scales endlessly. You don't have to rewrite anything and it provides a nice separation layer between your system components.

When starting out, you can easily run Kafka, DB, API on the same machine.

Re: Kafka is Fast – I'll use Postgres

#340

Post by [deleted] »

dxxvi wrote: post #337
saberience wrote: post #306

Earlier quoted context omitted.

Yes but the practical reality of it is it can be used exactly the same way as you would do a queue and you can make it work just as well as any MQ based system. I know this as I moved from a RabbitMQ system to Kafka for additionally scalability requirements and it worked perfectly. So sure "technically" it's not a queue, but in reality its used as a queue for 1000s of companies around the world for huge production wo…

Is it true that a message from a queue will disappear after it is consumed successfully? If yes, at this moment, how do you make kafka topics work as queues?

[deleted]
Post reply on HN