Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

231–240 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#231

Post by name_nick_sex_m »

vbezhenar wrote: post #215
grogers wrote: post #212

Earlier quoted context omitted.

You can fill in a noop for sequence number 123 after a timeout. You also need to be able to kill old transactions so that the transaction which was assigned 123 isn't just chilling out (which would block writing the noop). Another approach which I used in the past was to assign sequence numbers after committing. Basically a separate process periodically scans the set of un-sequenced rows, applies any application defi…

My approach is: select max(id), and commit with id=max(id)+1. If commit worked, then all good. If commit failed because of unique index violation, repeat the transaction from the beginning. I think it should work correctly with proper transaction isolation level.

Does the additional read query cause concern? Or mostly this is ok? (i'm sure the answer depends on scale)

Re: Kafka is Fast – I'll use Postgres

#232

Post by name_nick_sex_m »

vbezhenar wrote: post #27

How do you implement "unique monotonically-increasing offset number"? Naive approach with sequence (or serial type which uses sequence automatically) does not work. Transaction "one" gets number "123", transaction "two" gets number "124". Transaction "two" commits, now table contains "122", "124" rows and readers can start to process it. Then transaction "one" commits with its "123" number, but readers already past "…

Funnily enough, I was just designing a queue exactly this way, thanks for catching this. (chat GPT meanwhile was assuring me the approach was airtight)

Re: Kafka is Fast – I'll use Postgres

#233

Post by AtlasBarfed »

agentultra wrote: post #6

You have to be careful with the approach of using Postgres for everything. The way it locks tables and rows and the serialization levels it guarantees are not immediately obvious to a lot of folks and can become a serious bottle-neck for performance-sensitive workloads. I've been a happy Postgres user for several decades. Postgres can do a lot! But like anything, don't rely on maxims to do your engineering for you.

Postgres is just fantastic software.

But anytime you treat a database, or a queue, like a black box dumpster, problems will ensue.

Re: Kafka is Fast – I'll use Postgres

#234

Post by marcosdumay »

Groxx wrote: post #127
marcosdumay wrote: post #112

Earlier quoted context omitted.

Take a look on how to implement access control over GraphQL requests. It's useless for anything that isn't public data (at least public for your entire network). And yes, you don't want to use it for public APIs. But if you have private APIs that are so complex that you need a query language, and still want use those over web services, you are very likely doing something really wrong.

I'm honestly not seeing much here that isn't identical to almost all other general purpose RPC systems: https://graphql.org/learn/authorization/ "check that the user matches the data they're requesting by comparing the context and request field by hand" is ultra common - there are some real benefits to having authorization baked into the language, but it seems very rare in practice (which is part of why it's often fl…

> identical to almost all other general purpose RPC systems

The problem is that GraphQL doesn't behave like all other general purpose RPC systems. As a rule, authorization does not work on the same abstraction level as GraphQL.

And that explanation you quoted is disingenuous, because GraphQL middleware and libraries don't usually export places where you can do anything by hand.

Re: Kafka is Fast – I'll use Postgres

#235

Post by AtlasBarfed »

javier2 wrote: post #211
agentultra wrote: post #6

You have to be careful with the approach of using Postgres for everything. The way it locks tables and rows and the serialization levels it guarantees are not immediately obvious to a lot of folks and can become a serious bottle-neck for performance-sensitive workloads. I've been a happy Postgres user for several decades. Postgres can do a lot! But like anything, don't rely on maxims to do your engineering for you.

Postgres doesnt scale into oblivion, but it can take some serious chunks of data once you start batching and making sure a every operation only touches single row with no transactions needed.

And then you are 99% of the way to Cassandra.

Of course the other 99% is the remaining 1%.

Re: Kafka is Fast – I'll use Postgres

#236

Post by hunterpayne »

vbezhenar wrote: post #27

How do you implement "unique monotonically-increasing offset number"? Naive approach with sequence (or serial type which uses sequence automatically) does not work. Transaction "one" gets number "123", transaction "two" gets number "124". Transaction "two" commits, now table contains "122", "124" rows and readers can start to process it. Then transaction "one" commits with its "123" number, but readers already past "…

The "unique monotonically-increasing offset number" use case works just fine. I need a unique sequence number in ascending order doesn't (your problem). Why you need two queue to share the same sequence object is your problem I think.

Another way to speed it up is to grab unique numbers in batches instead of just getting them one at a time. No idea why you want your numbers to be in absolute sequence. That's hard in a distributed system. Probably best to relax that constraint and find some other way to track individual pieces of data. Or even better, find a way so you don't have to track individual rows in a distributed system.

Re: Kafka is Fast – I'll use Postgres

#237

Post by skunkworker »

TexanFeller wrote: post #230
ozim wrote: post #210

Earlier quoted context omitted.

Periodic polling of a DB gets bad pretty quick, queues are much better even on small scale. But then distributed queue is most likely not needed until you hit really humongous scale.

Maybe in the past this was true, or if you’re using an inferior DB. I know first hand that a Postgres table can work great as a queue for many millions of events per day processed by thousands of workers polling for work from it concurrently. With more than a few hundred concurrent pollers you might want a service, or at least a centralized connection pool in front of it though.

Millions of events per day is still in the small queue category in my book. Postgres LISTEN doesn't scale, and polling on hot databases can suddenly become more difficult, as you're having to throw away tuples regularly.

10 message/s is only 860k/day. But in my testing (with postgres 16) this doesn't scale that well when you are needing tens to hundreds of millions per day. Redis is much better than postgres for that (for a simple queue), and beyond that kafka is what I would choose in you're in the low few hundred million.

Re: Kafka is Fast – I'll use Postgres

#238

Post by skunkworker »

agentultra wrote: post #6

You have to be careful with the approach of using Postgres for everything. The way it locks tables and rows and the serialization levels it guarantees are not immediately obvious to a lot of folks and can become a serious bottle-neck for performance-sensitive workloads. I've been a happy Postgres user for several decades. Postgres can do a lot! But like anything, don't rely on maxims to do your engineering for you.

I wish postgres would add a durable queue like data structure. But trying to make a durable queue that can scale beyond what a simple redis instance can do starts to run into problems quickly.

Also, LISTEN/NOTIFY do not scale, and they introduce locks in areas you aren't expecting - https://news.ycombinator.com/item?id=44490510

Re: Kafka is Fast – I'll use Postgres

#240

Post by pram »

rjurney wrote: post #203

One bad message in a Kafka queue and guess what? The entire queue is down because it kills your workers over and over. To fix it? You have to resize the queue to zero, which means losing requests. This KILLS me. Jay Kreps says there is no reason it can't be fixed, but it never had been and this infuriates me because it happens so often :)

You can modify a consumer groups offset to any value JFYI, so you really don’t need to purge the topic. You can just start after the bad message.
Post reply on HN