Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is Fast – I'll use Postgres

topicpartition.io

Post reply on HN
Print view View original

301–310 of 412 posts

Re: Kafka is Fast – I'll use Postgres

#301

Post by sublimefire »

dagss wrote: post #268

I really believe this is the way: Event log tables in SQL. I have been doing it a lot. A downside is the lack of tooling client side. For many using Kafka is worth it simply for the tooling in libraries consumer side. If you just want to write an event handler function there is a lot of boilerplate to manage around it. (Persisting read cursors etc) We introduced a company standard for one service pulling events from…

With the advent of tools like llms in editors, it is now viable to create clients and solve these gaps quite easily. It feels like the next low hanging fruit to do in many places not client friendly enough.

Re: Kafka is Fast – I'll use Postgres

#302

Post by BerislavLopac »

EdwardDiego wrote: post #271
munchbunny wrote: post #119

My general opinion, off the cuff, from having worked at both small (hundreds of events per hour) and large (trillions of events per hour) scales for these sorts of problems: 1. Do you really need a queue? (Alternative: periodic polling of a DB) 2. What's your event volume and can it fit on one node for the foreseeable future, or even serverless compute (if not too expensive)? (Alternative: lightweight single-process…

Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement for an MQ, and it's such the wrong way to think about it.

To be fair, any (immutable) data structure that includes the creation timestamp can be a queue. It might not be a good queue, but it can be used as one.

Re: Kafka is Fast – I'll use Postgres

#303

Post by sublimefire »

abtinf wrote: post #260
spectraldrift wrote: post #257

> Should You Use Postgres? Most of the time - yes This made me wonder about a tangential statistic that would, in all likelihood, be impossible to derive: If we looked at all database systems running at any given time, what proportion does each technology represent (e.g., Postgres vs. MySQL vs. [your favorite DB])? You could try to measure this in a few ways: bytes written/read, total rows, dollars of revenue served,…

SQLite likely dominates all other databases combined on the metrics you mentioned, I would guess by at least an order of magnitude. Server side. Client side. iOS, iPad, Mac apps. Uses in every field. Uses in aerospace. Just think for a moment that literally every photo and video taken on every iPhone (and I would assume android as well) ends up stored (either directly or sizable amounts of metadata) in a SQLite db.

Yes it seems like it is absent in this discussion but maybe it should have been “it” the whole time as a default option. I wonder if it could attain similar throughput numbers; bet the article would feel slightly sarcastic then though

Re: Kafka is Fast – I'll use Postgres

#304

Post by 59nadir »

procaryote wrote: post #214
lumost wrote: post #190

Earlier quoted context omitted.

I suspect the common issue with small scale projects is that it's not atypical for the engineers involved to perform a joint optimization of "what will work well for this project", and "what will work well at my next project/job." Particularly in startups where the turnover/employer stability is poor - this is the optimal action for the engineers involved. Unless employees expect that their best rewards are from maki…

This is something to catch in hiring and performance evaluation. Hire people who don't build things to pad their own CVs, tell them to stop if you failed, fire them if that failed

This is a great way to get only people who basically can't build anything.

Re: Kafka is Fast – I'll use Postgres

#305

Post by 59nadir »

TexanFeller wrote: post #230
ozim wrote: post #210

Earlier quoted context omitted.

Periodic polling of a DB gets bad pretty quick, queues are much better even on small scale. But then distributed queue is most likely not needed until you hit really humongous scale.

Maybe in the past this was true, or if you’re using an inferior DB. I know first hand that a Postgres table can work great as a queue for many millions of events per day processed by thousands of workers polling for work from it concurrently. With more than a few hundred concurrent pollers you might want a service, or at least a centralized connection pool in front of it though.

This "per hour" and "per day" business has to end. No one cares about "per day" and it makes it much harder to see the actual talked about load on a system. The thing that matters is "per second", so why not talk about exactly that? Load is something immediate, it's not a "per day" thing.

If someone is talking about per day numbers or per month numbers they're likely doing it to have the numbers sound more impressive and to make it harder to see how few X per second they actually handled. 11 million events per day sounds a whole lot more impressive than 128 events per second, but they're the same thing and only the latter usually matters in these types of discussions.

Re: Kafka is Fast – I'll use Postgres

#306

Post by saberience »

EdwardDiego wrote: post #271
munchbunny wrote: post #119

My general opinion, off the cuff, from having worked at both small (hundreds of events per hour) and large (trillions of events per hour) scales for these sorts of problems: 1. Do you really need a queue? (Alternative: periodic polling of a DB) 2. What's your event volume and can it fit on one node for the foreseeable future, or even serverless compute (if not too expensive)? (Alternative: lightweight single-process…

Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement for an MQ, and it's such the wrong way to think about it.

Yes but the practical reality of it is it can be used exactly the same way as you would do a queue and you can make it work just as well as any MQ based system. I know this as I moved from a RabbitMQ system to Kafka for additionally scalability requirements and it worked perfectly.

So sure "technically" it's not a queue, but in reality its used as a queue for 1000s of companies around the world for huge production workloads which no MQ system can support.

Re: Kafka is Fast – I'll use Postgres

#308

Post by hobs »

sarchertech wrote: post #273
DelaneyM wrote: post #213

Earlier quoted context omitted.

My suggestion would be even simpler: MQTT -> Postgres (+ S3 for archive) > 1. my "fear" would be that if I use the same Postgres for the queue and for my business database... This is a feature, not a bug. In this way you can pair the handling of the message with the business data changes which result in the same transaction. This isn't quite "exactly-once" handling, but it's really really close! > 2. also that since…

> This is a feature, not a bug. In this way you can pair the handling of the message with the business data changes which result in the same transaction. That’s a particularly nasty trap. Devs will start using this everywhere and it makes it very hard to move this beyond Postgres when you need to. I’d keep a small transactional outbox for when you really need it and encourage devs to use it only when absolutely neces…

The problem is either you have this feature or you dont, misusing it is another problem. Not having a feature sucks, and most distributed databases will even give you options for consistent (slow ass) reads.

Re: Kafka is Fast – I'll use Postgres

#309

Post by hamandcheese »

BerislavLopac wrote: post #302
EdwardDiego wrote: post #271

Earlier quoted context omitted.

Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement for an MQ, and it's such the wrong way to think about it.

To be fair, any (immutable) data structure that includes the creation timestamp can be a queue. It might not be a good queue, but it can be used as one.

On this note... has anyone here used object storage directly as a queue? How did it go?

You can make a bucket immutable, and entries have timestamps. I don't think any cloud provider makes claims about the accuracy or monoatomicity of these timestamps, so you would merely get an ordering, not necessarily the ordering in which things truly occurred. But I have use cases where that is fine.

I believe with a clever naming scheme and cooperating clients it could be made to work.

Post reply on HN