Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

311–320 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#311

Post by dvhh »

hamandcheese wrote: post #309
BerislavLopac wrote: post #302

Earlier quoted context omitted.

To be fair, any (immutable) data structure that includes the creation timestamp can be a queue. It might not be a good queue, but it can be used as one.

On this note... has anyone here used object storage directly as a queue? How did it go? You can make a bucket immutable, and entries have timestamps. I don't think any cloud provider makes claims about the accuracy or monoatomicity of these timestamps, so you would merely get an ordering, not necessarily the ordering in which things truly occurred. But I have use cases where that is fine. I believe with a clever nami…

Once used object storage as queue, you can implement queue semantic at the application level, with one object per entry.

But the application was fairly low volume in Data and usage, So eventual consistency and capacity was not an issue. And yes timestamp monotonicity is not guaranteed when multiple client upload at the same time so unique id was given to each client at startup and used for to add guarantee of entries name. Metadata and prefix were used to indicate state of object during processing.

Not ideal, but it was cheaper that a DB or a dedicated MQ. The application did not last, but would try again the approach if adapted to stuation.

Re: Kafka is Fast – I'll use Postgres

#312

Post by bostik »

drdaeman wrote: post #246
munchbunny wrote: post #119

My general opinion, off the cuff, from having worked at both small (hundreds of events per hour) and large (trillions of events per hour) scales for these sorts of problems: 1. Do you really need a queue? (Alternative: periodic polling of a DB) 2. What's your event volume and can it fit on one node for the foreseeable future, or even serverless compute (if not too expensive)? (Alternative: lightweight single-process…

> Do you really need a queue? (Alternative: periodic polling of a DB) In my experience it’s not the reads, but the writes that are hard to scale up. Reading is cheap and can be sometimes done off a replica. Writing to a PostgreSQL at high sustained rate requires careful tuning and designs. A stream of UPDATEs can be very painful, INSERTs aren’t cheap, and even a batched COPY blocks can be tricky.

Plus of course you can take out the primary even with a read from a replica. It's not a trivial feat, but you can achieve it with the combination of streaming replication and an hours-long read from the replica for massive analytical workloads. For large reads Postgres will create temporary tables as needed, and when those in the replica end up far enough, the cascading effect through replication backpressure will cause primary to block further writes from getting through...

The scars from that kind of outage will never truly heal.

Re: Kafka is Fast – I'll use Postgres

#313

Post by wosined »

antonvs wrote: post #186
wosined wrote: post #39

Earlier quoted context omitted.

I am the third pole: 3. Everything we have currently sucks and what is new will suck for some hitherto unknown reason.

If you choose wisely, things should suck less overall as you move forward. That's kind of the overall goal, otherwise we'd all still be toggling raw machine code into machines using switches.

Computers got faster, software is not so straightforward. I don't even know why a text webpage needs 100mb of memory to render and display.

Re: Kafka is Fast – I'll use Postgres

#314

Post by name_nick_sex_m »

1oooqooq wrote: post #244
name_nick_sex_m wrote: post #232

Earlier quoted context omitted.

Funnily enough, I was just designing a queue exactly this way, thanks for catching this. (chat GPT meanwhile was assuring me the approach was airtight)

you're really trying to vibe architect?

Gotta make a living somehow

Re: Kafka is Fast – I'll use Postgres

#315

Post by LinXitoW »

Isn't one gigantic advantage with Postgres the ACID part?

It seems to me that the hardest part of going for a MQ/distributed log like Kafka is re-working existing code to now handle the lack of ACID stuff. Things that are trivial with Postgres, like exactly once delivery, are huge undertakings without ACID.

Personally, I don't have much experience with this, so maybe I'm just missing something?

Re: Kafka is Fast – I'll use Postgres

#316

Post by coldtea »

>The claim is that it handles 80%+ of their use cases with 20% of the development effort. (Pareto Principle)

The Pareto principle is not some guarantee applicable to everything and anything saying that any X will handle 80% of some other thing's use cases with 20% the effort.

One can see how irrelevant its invocation is if we reverse: does Kafka also handle 80% of what Postgres does with 20% the effort? If not, what makes Postgres especially the "Pareto 80%" one in this comparison? Did Vilfredo Pareto had Postgres specifically in mind when forming the principle?

Pareto principle concerns situations where power-law distributions emerge. Not arbitrary server software comparisons.

Just say Postgres covers a lot of use cases people mindlessly go to shiny new software for that they don't really need, and is more battled tested, mature, and widely supported.

The Pareto principle is a red herring.

Re: Kafka is Fast – I'll use Postgres

#317

Post by animuchan »

jghn wrote: post #241
lumost wrote: post #190

Earlier quoted context omitted.

I suspect the common issue with small scale projects is that it's not atypical for the engineers involved to perform a joint optimization of "what will work well for this project", and "what will work well at my next project/job." Particularly in startups where the turnover/employer stability is poor - this is the optimal action for the engineers involved. Unless employees expect that their best rewards are from maki…

What I've found to be even more common than resume driven development has been people believing that they either have or will have "huge scale". But the problem is that their goal posts are off by a few orders of magnitude and they will never, ever have the sort of scale required for these types of tools.

LLM solves this by meeting devs in the middle: the vibe coded DB schema, coupled with agentically-made application code, makes even 20,000 records a "huge scale".

Re: Kafka is Fast – I'll use Postgres

#318

Post by hamandcheese »

dvhh wrote: post #311
hamandcheese wrote: post #309

Earlier quoted context omitted.

On this note... has anyone here used object storage directly as a queue? How did it go? You can make a bucket immutable, and entries have timestamps. I don't think any cloud provider makes claims about the accuracy or monoatomicity of these timestamps, so you would merely get an ordering, not necessarily the ordering in which things truly occurred. But I have use cases where that is fine. I believe with a clever nami…

Once used object storage as queue, you can implement queue semantic at the application level, with one object per entry. But the application was fairly low volume in Data and usage, So eventual consistency and capacity was not an issue. And yes timestamp monotonicity is not guaranteed when multiple client upload at the same time so unique id was given to each client at startup and used for to add guarantee of entries…

The application I'm interested in is a log-based artifact registry. Volume would be very low. Much more important is the immutability and durability of the log.

I was thinking that writes could be indexed/prefixed into timestamp buckets according to the clients local time. This can't be trusted, of course. But the application consumers could detect and reject any writes whose upload timestamp exceeds a fixed delta from the timestamp bucket it was uploaded to. That allows for arbitrary seeking to any point on the log.

Re: Kafka is Fast – I'll use Postgres

#319

Post by speed_spread »

EdwardDiego wrote: post #286
codeflo wrote: post #280

Earlier quoted context omitted.

Do you mean this in the sense that listeners don't remove messages, as one would expect from a queue data structure?

Exactly. There's no concept in Kafka (yet...) of "acking" or DLQs, Kafka is very good at what it does by being deliberately stupid, it knows nothing about your messages or who has consumed them and who hasn't. That was all deliberately pushed onto consumers to manage to achieve scale.

Kafka is the MongoDB of sequential storage. Built for webscale and then widely adopted based on impressive single-metric numbers without regard for actual fitness to purpose in smaller operations. Fortunately it always what reliable enough.

I believe RabbitMQ is much more balanced and is closer to what people expect from a high level queueing/pubsub system.

Re: Kafka is Fast – I'll use Postgres

#320

Post by baq »

bostik wrote: post #312
drdaeman wrote: post #246

Earlier quoted context omitted.

> Do you really need a queue? (Alternative: periodic polling of a DB) In my experience it’s not the reads, but the writes that are hard to scale up. Reading is cheap and can be sometimes done off a replica. Writing to a PostgreSQL at high sustained rate requires careful tuning and designs. A stream of UPDATEs can be very painful, INSERTs aren’t cheap, and even a batched COPY blocks can be tricky.

Plus of course you can take out the primary even with a read from a replica. It's not a trivial feat, but you can achieve it with the combination of streaming replication and an hours-long read from the replica for massive analytical workloads. For large reads Postgres will create temporary tables as needed, and when those in the replica end up far enough, the cascading effect through replication backpressure will ca…

IME (...don't ask) it's easy enough if you forget to set idle in transaction timeout, though I haven't... tried... on replicas
Post reply on HN