Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

241–250 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#241

Post by jghn »

lumost wrote: post #190
munchbunny wrote: post #119

My general opinion, off the cuff, from having worked at both small (hundreds of events per hour) and large (trillions of events per hour) scales for these sorts of problems: 1. Do you really need a queue? (Alternative: periodic polling of a DB) 2. What's your event volume and can it fit on one node for the foreseeable future, or even serverless compute (if not too expensive)? (Alternative: lightweight single-process…

I suspect the common issue with small scale projects is that it's not atypical for the engineers involved to perform a joint optimization of "what will work well for this project", and "what will work well at my next project/job." Particularly in startups where the turnover/employer stability is poor - this is the optimal action for the engineers involved. Unless employees expect that their best rewards are from maki…

What I've found to be even more common than resume driven development has been people believing that they either have or will have "huge scale". But the problem is that their goal posts are off by a few orders of magnitude and they will never, ever have the sort of scale required for these types of tools.

Re: Kafka is Fast – I'll use Postgres

#242

Post by stackskipton »

loftsy wrote: post #199
singron wrote: post #113

Earlier quoted context omitted.

Kafka also doesn't give you all those things. E.g. there is no automatic dead-lettering, so a consumer that throws an exception will endlessly retry and block all progress on that partition. Kafka only stores bytes, so schema is up to you. Exactly-once is good, but there are some caveats (you have to use kafka transactions, which are significantly different than normal operation, and any external system may observe a…

Very interesting. I need a durable queue but not indefinitely. Max a couple of hours. What I want is Google PubSub but open source so I can self host.

Small size, Beanstalkd (https://beanstalkd.github.io/) can get you pretty far.

Larger, RabbitMQ can handle some pretty good workloads.

Re: Kafka is Fast – I'll use Postgres

#243

Post by javier2 »

AtlasBarfed wrote: post #235
javier2 wrote: post #211

Earlier quoted context omitted.

Postgres doesnt scale into oblivion, but it can take some serious chunks of data once you start batching and making sure a every operation only touches single row with no transactions needed.

And then you are 99% of the way to Cassandra. Of course the other 99% is the remaining 1%.

Nearly true, but you dont need to run a cassandra cluster to ship your 3k msg/sec and you can take smaller locks if you have a small number of senders that delete sent messages and send in chunks

Re: Kafka is Fast – I'll use Postgres

#244

Post by 1oooqooq »

name_nick_sex_m wrote: post #232
vbezhenar wrote: post #27

How do you implement "unique monotonically-increasing offset number"? Naive approach with sequence (or serial type which uses sequence automatically) does not work. Transaction "one" gets number "123", transaction "two" gets number "124". Transaction "two" commits, now table contains "122", "124" rows and readers can start to process it. Then transaction "one" commits with its "123" number, but readers already past "…

Funnily enough, I was just designing a queue exactly this way, thanks for catching this. (chat GPT meanwhile was assuring me the approach was airtight)

you're really trying to vibe architect?

Re: Kafka is Fast – I'll use Postgres

#245

Post by tele_ski »

brikym wrote: post #239

If you don't mind Redis then use Redis Streams. It gives you an eventlog without worrying about postgres performance issues and has consumer groups.

Been using valkey streams recently and loving it. Took a bit to understand how to to properly use it but now that I've figured it out I'd highly recommend trying it. It's very easy to setup and get going and just works.

Re: Kafka is Fast – I'll use Postgres

#246

Post by drdaeman »

munchbunny wrote: post #119

My general opinion, off the cuff, from having worked at both small (hundreds of events per hour) and large (trillions of events per hour) scales for these sorts of problems: 1. Do you really need a queue? (Alternative: periodic polling of a DB) 2. What's your event volume and can it fit on one node for the foreseeable future, or even serverless compute (if not too expensive)? (Alternative: lightweight single-process…

> Do you really need a queue? (Alternative: periodic polling of a DB)

In my experience it’s not the reads, but the writes that are hard to scale up. Reading is cheap and can be sometimes done off a replica. Writing to a PostgreSQL at high sustained rate requires careful tuning and designs. A stream of UPDATEs can be very painful, INSERTs aren’t cheap, and even a batched COPY blocks can be tricky.

Re: Kafka is Fast – I'll use Postgres

#247

Post by abtinf »

skunkworker wrote: post #238
agentultra wrote: post #6

You have to be careful with the approach of using Postgres for everything. The way it locks tables and rows and the serialization levels it guarantees are not immediately obvious to a lot of folks and can become a serious bottle-neck for performance-sensitive workloads. I've been a happy Postgres user for several decades. Postgres can do a lot! But like anything, don't rely on maxims to do your engineering for you.

I wish postgres would add a durable queue like data structure. But trying to make a durable queue that can scale beyond what a simple redis instance can do starts to run into problems quickly. Also, LISTEN/NOTIFY do not scale, and they introduce locks in areas you aren't expecting - https://news.ycombinator.com/item?id=44490510

SKIP LOCKED doesn't work for your use case?

Re: Kafka is Fast – I'll use Postgres

#250

Post by altcognito »

loire280 wrote: post #180
altcognito wrote: post #153

Earlier quoted context omitted.

This doesn't even make sense. How do you know what the network links or the other bottlenecks are like? There are a grandiose number of assumptions being made here.

There is a finite and relatively narrow range of ratios of CPU, memory, and network throughput in both modern cloud offerings and bare hardware configurations. Obviously it's possible to build, for example, a machine with 2 cores, a 10Gbps network link, and a single HDD that would falsify my statement.

But the workload matters. Even the comment in the article doesn't completely make sense for me in that way -- if your workload is 50 operations per byte transferred versus 5000 operations per byte transferred, there is a considerable difference in hardware requirements.
Post reply on HN