My general opinion, off the cuff, from having worked at both small (hundreds of events per hour) and large (trillions of events per hour) scales for these sorts of problems: 1. Do you really need a queue? (Alternative: periodic polling of a DB) 2. What's your event volume and can it fit on one node for the foreseeable future, or even serverless compute (if not too expensive)? (Alternative: lightweight single-process…
> 1. Do you really need a queue? I'm a java dev and maybe my projects are about big integrations, but I've always needed queue like constructs and polling from a db was almost always a headache, especially with multiple consumers and publishers. Sure it can be done, and in many projects we do have cron-jobs on different pods -- not a global k8s cron-job, but legacy cron jobs and it works fine. Kafka does not YET supp…
Kafka is Fast – I'll use Postgres
381–390 of 412 posts
Re: Kafka is Fast – I'll use Postgres
#382Earlier quoted context omitted.
> you can make it work just as well as any MQ based system you really can't. getting per-message acks, dynamically scaling competing consumers without having to repartition while retaining ordering, etc. requires a ton of hacks like client side tracking / building your own storage on top of offset metadata / etc.. and you still won't have all of the features actual message queues provide. to make it worse, there is v…
There is a whole KIP that is “preview” on Kafka 4.1 to handle this use case natively: https://cwiki.apache.org/confluence/plugins/servlet/mobile?c... Note: I haven’t had a chance to test it out in anger personally yet.
yep i mentioned that in another thread below. very excited to see it added, but it will be some time until its production ready / they implement the full feature set
Re: Kafka is Fast – I'll use Postgres
#383Earlier quoted context omitted.
I think the intention of the original license was to make the software unpalatable to SaaS vendors who want to keep their changes proprietary, not unpalatable to enterprises in general.
Rightly or wrongly, large companies are very averse to using AGPL software even if it would cause them very little additional burden to comply with the AGPL. Lots of projects use this cynically to help sell proprietary licenses (the proof of this is self-evident -- many such projects have CLAs and were happy to switch to a proprietary license that is even less favourable to enterprises than the AGPL as soon as it was…
I haven't actually talked with Henry Poole about the subject, but I'm pretty sure that was not his intent when he wrote it.
Re: Kafka is Fast – I'll use Postgres
#384Earlier quoted context omitted.
Another good item to consider: n) Do you really need S3? is it cheaper than NFS storage on a compute node with a large disk? There are many cases where S3 is absolutely cheaper though.
In my experience NFS is always the wrong thing to use. Your application think it's a normal disk but it isn't, so you get no timeouts, no specific errors for network issues and extremely expensive calls camouflage as quick FS ops (was any file modified in this folder ? I'll just loop over them using my standard library nice FS utilities). And you don't get atomic ops outside of mv, invalidation and caching are compli…
Safe to assume s3fs and smb have similar issues?
I would think something like NFS is best suited for an actual file instead of blob you're serializing using a file system api?
Re: Kafka is Fast – I'll use Postgres
#385I'm solidly in camp 2, the "common sense" camp that doesn't care about buzzwords. That said, I don't consider running Kafka to be a headache. I work at a mid-sized company, processing billions of Kafka events per day and it's never been a problem, even locally when I'm processing hundreds of events per day. You set it up, forget about it, and it scales endlessly. You don't have to rewrite anything and it provides a n…
> processing billions of Kafka events per day Except that the burden is on all clients to coordinate to avoid processing an event more than once since Kakfa is a brainless invention just dumping data forever into a serial log.
That is called a 'consumer group' which has been a part of Kafka for 15 years.
The author is suggesting to avoid this solution and roll your own instead.
Re: Kafka is Fast – I'll use Postgres
#386Earlier quoted context omitted.
Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement for an MQ, and it's such the wrong way to think about it.
Yes but the practical reality of it is it can be used exactly the same way as you would do a queue and you can make it work just as well as any MQ based system. I know this as I moved from a RabbitMQ system to Kafka for additionally scalability requirements and it worked perfectly. So sure "technically" it's not a queue, but in reality its used as a queue for 1000s of companies around the world for huge production wo…
Don't you always need a database after reading events from Kafka to deduplication?
So the competing solutions are: PostgreSQL or Kafka+PostgreSQL
Kafka does provide some extrs there, handling load spikes, more clients that PG can handle natively and resilience to some DB downtime. But is it worth the complexity, in most cases no.
Re: Kafka is Fast – I'll use Postgres
#387Earlier quoted context omitted.
This is so accurate. I’ve looked in wonder at how someone is maxing out an r6i.32xlarge MySQL DB, when I have ran 4x the workload on an r6i.12xlarge. Schema design and query design will make or break your app’s ability to scale without skyrocketing the bill, it’s as simple as that.
any blogs/books you'd recommend on schema & query design? it honestly surprises me that these coding-focused models can't look at a schema; look at how data is being queried; reason about the use case for the data; and help prioritize solving for the most likely bottlenecks to scaling the underlying data services.
https://learn.microsoft.com/en-us/sql/relational-databases/s...
This one is a classic for MSSQL, most of it is applicable on postgres.
Re: Kafka is Fast – I'll use Postgres
#388Earlier quoted context omitted.
Why can't you? In my experience, scaling Kafka was far easier than scaling our RabbitMQ cluster. We started running into issues when our RabbitMQ cluster hit 25k TPS, our Kafka cluster of equivalent resources didn't break a sweat at 500k TPS.
What sort of systems do you work on to require this kind of traffic volume? I've worked on one project that I'd consider relatively high volume (UK Post Office Horizon Online) and we were only targeting 500 TPS.
Think extremely popular multiplayer videogames. We used these systems for all the backend infra, logins, logouts, chat messages, purchases.
We often had millions of players online at a given moment which means lots of transactions!
Re: Kafka is Fast – I'll use Postgres
#389Earlier quoted context omitted.
This is so accurate. I’ve looked in wonder at how someone is maxing out an r6i.32xlarge MySQL DB, when I have ran 4x the workload on an r6i.12xlarge. Schema design and query design will make or break your app’s ability to scale without skyrocketing the bill, it’s as simple as that.
any blogs/books you'd recommend on schema & query design? it honestly surprises me that these coding-focused models can't look at a schema; look at how data is being queried; reason about the use case for the data; and help prioritize solving for the most likely bottlenecks to scaling the underlying data services.
I like http://www.redbook.io/
Re: Kafka is Fast – I'll use Postgres
#390Earlier quoted context omitted.
> This is a feature, not a bug. In this way you can pair the handling of the message with the business data changes which result in the same transaction. That’s a particularly nasty trap. Devs will start using this everywhere and it makes it very hard to move this beyond Postgres when you need to. I’d keep a small transactional outbox for when you really need it and encourage devs to use it only when absolutely neces…
The problem is either you have this feature or you dont, misusing it is another problem. Not having a feature sucks, and most distributed databases will even give you options for consistent (slow ass) reads.
If you have a database that supports transactions and something like skip locked, you always have option of building a transactional outbox when you need it.