Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka at the low end: how bad can it get?

broot.ca

Post reply on HN
Print view View original

61–70 of 144 posts

Re: Kafka at the low end: how bad can it get?

#61

Post by winrid »

stickfigure wrote: post #58
pritambaral wrote: post #55

Earlier quoted context omitted.

I hit 60k per second in 2020 on a 2-core, 100GB SSD installation of PG on GCP. And "tuning" PG is way easier than any dedicated queueing system I've seen. Does there exist a dedicated queueing system with an equivalent to EXPLAIN (ANALYZE)?

If that's true, you managed to do much better than these folks: https://softwaremill.com/mqperf/ Maybe you should write a letter?

It's possible the person you're replying to wasn't using replication, so it's entirely different. Those folks also used "synchronous_commit is set to remote_write" which will have a performance impact

Re: Kafka at the low end: how bad can it get?

#62

Post by ofrzeta »

bassp wrote: post #49
NovemberWhiskey wrote: post #14

Kafka for small message volumes is one of those distinct resume-padding architectural vibes.

I use Kafka for a low-message-volume use case because it lets my downstream consumers replay messages… but yeah in most cases, it’s over kill

That was also a use case for me. However at some point I replaced Kafka with Redpanda.

Re: Kafka at the low end: how bad can it get?

#63

Post by snotrockets »

araes wrote: post #34

Having never actually used this platform before, does anybody know why they named it Kafka, with all the horrible meanings? Per Wiktionary, Kafkaesque: [1] 1. "Marked by a senseless, disorienting, often menacing complexity." 2. "Marked by surreal distortion and often a sense of looming danger." 3. "In the manner of something written by Franz Kafka." (like the software language was written by Franz Kafka) Example: Met…

It was named so based on the Idea is that like the author (who the term "Kafkesque" is coined after), Apache Kafka is a prolific writer.

Re: Kafka at the low end: how bad can it get?

#64

Post by FearNotDaniel »

NovemberWhiskey wrote: post #14

Kafka for small message volumes is one of those distinct resume-padding architectural vibes.

That’s almost certainly true, but at least part of the problem (not just Kafka but RDD tech in general) is that project home pages, comments like this and “Learn X in 24 hours” books/courses rarely spell out how to clearly determine if you have an appropriate use case at an appropriate scale. “Use this because all the cool kids are using it” affects non-tech managers and investors just as much as developers with no architectural nous, and everyone with a SQL connection and an API can believe they have “big data” if they don’t have a clear definition of what big data actually is.

Re: Kafka at the low end: how bad can it get?

#65

Post by hhh »

rogerthis wrote: post #24
sea-gold wrote: post #7

Earlier quoted context omitted.

NATS https://docs.nats.io/nats-concepts/overview/compare-nats

I'd wish NATS were more popular. It feels it lacks some real big sponsors $$$.

Scaling with NATS seems weird. I like what i’ve seen with others using it though

Re: Kafka at the low end: how bad can it get?

#66

Post by hiAndrewQuinn »

NovemberWhiskey wrote: post #14

Kafka for small message volumes is one of those distinct resume-padding architectural vibes.

Apt time to mention the classic "Command-line Tools can be 235x Faster than your Hadoop Cluster", for those who may have not yet read it.

https://adamdrake.com/command-line-tools-can-be-235x-faster-...

Re: Kafka at the low end: how bad can it get?

#67

Post by javaunsafe2019 »

Joel_Mckay wrote: post #46
kvakerok wrote: post #30

Earlier quoted context omitted.

You haven't seen the worst of it. We had to implement a whole kafka module for a SCADA system because Target already had unrelated kafka infrastructure. Instead of REST API or anything else sane (which was available), ultra low volume messaging is now done by JSON objects wrapped in kafka. Peak incompetence.

We did something similar using RabbitMQ with bson over AMQP, and static message routing. Anecdotally, the design has been very reliable for over 6 years with very little maintenance on that part of the system, handles high-latency connection outage reconciliation, and new instances are cycled into service all the time. Mostly people that ruminate on naive choices like REST/HTTP2/MQTT will have zero clue how the probl…

I don’t know why but I could wear you are German (and old)

Re: Kafka at the low end: how bad can it get?

#68

Post by atmosx »

NovemberWhiskey wrote: post #14

Kafka for small message volumes is one of those distinct resume-padding architectural vibes.

Oh no!

Let’s be real: teams come to the infra team asking for a queue system. They give their requirements, and you—like a responsible engineer—suggest a more capable queue to handle their needs more efficiently. But no, they want Kafka. Kafka, Kafka, Kafka. Fine. You (meaning an entire team) set up Kafka clusters across three environments, define SLIs, enforce SLOs, make sure everything is production-grade.

Then you look at the actual traffic: 300kb/s in production. And right next to it? A RabbitMQ instance happily chugging along at 200kb/s.

You sit there, questioning every decision that led you to this moment. But infra isn’t the decision-maker. Sometimes, adding unnecessary complexity just makes everyone happier. And no, it’s not just resume-padding… probably.

Re: Kafka at the low end: how bad can it get?

#70

Post by petepete »

stackskipton wrote: post #41
jszymborski wrote: post #3

What do people recommend? Especially for low levels of load, that doesn't require that the dispatcher and consumer are written in the same language.

Database is good recommendation. Also give a shoutout to Beanstalkd ( https://beanstalkd.github.io/ )

I love beanstalkd, used it for plenty over the years and it just flies through with no fuss. Plus, it's fully supported by Rails' ActiveJob.
Post reply on HN