11–20 of 102 posts
The anti-pattern here isn't Kafka, it's using Kafka for 7,500 messages/day. Making your whole system asynchronous for that level of load is the textbook definition of over-engineering.
The anti-pattern here isn't Kafka, it's using Kafka for 7,500 messages/day. Making your whole system asynchronous for that level of load is the textbook definition of over-engineering.
It seems a lot of the complaints weren't about kafka itself, but rather seemed to stem from internal communication problems. Custom kafka message headers could very well be custom http headers, and the problem is the same. Kafka is just coincidental. Looking at the volume though, kafka is overkill. They most likely could have just used the database and reaped the benefits of doing everything in a single transaction,…
does something like that exist ???
The anti-pattern here isn't Kafka, it's using Kafka for 7,500 messages/day. Making your whole system asynchronous for that level of load is the textbook definition of over-engineering.
Yes. For perspective, that's about one message every ten seconds.
For example, if you use Kafka or streaming solutions like Flink or Spark, you should change your mental model to (possibly large), (possibly resplayable) streams of events and look for simple ways to get these event streams going and good ways to consume them. And then you need to let the design push you where it wants you to go.
Like, at work, we recently had a discussion how it was so storage-expensive for a project to store all events of a day and how the query to count all of these events per tenant was taking so long. While they are using a streaming event processor in front of it. Like, what the hell - think in streams, tally up these events on the fly and persist that every hour?
Immediately starts to doubt OP's assumption/implementation of the "where it works great"
Then again am sure there maybe reasons/context/constraints unaware to us.
Eg - the ingestion is spiky, with the possibility of all 7500 in a few secs/minute, you would want to first make sure that the http traffic can scale before getting to the point where it can actually connect and push to the queue
Possible reason#2 - an intern who just finished up their first Kafka task; just got freed up
#3 - or this was the only infra available and a choice had to be made with the time available at hand
#4 - or this was an experiment to see for yourself
A majority may not agree with your view, but none of us really are in your shoes. So I applaud you for sharing your thoughts anyway.
Earlier quoted context omitted.
Yes. For perspective, that's about one message every ten seconds.
Sorry for the naivety/not obvious from the comments: is that too much or too little? (I've used RabbitMQ much more than Kafka.)
Earlier quoted context omitted.
Yes. For perspective, that's about one message every ten seconds.
Sorry for the naivety/not obvious from the comments: is that too much or too little? (I've used RabbitMQ much more than Kafka.)
Most businesses have no (hue hue) business doing event driven architecture. There is way too much overhead for local testing and overall complexity, especially when you want to properly handle errors.
"But, every developer should be able to set up their local." Yea, great, explain to the manual QA who may be amazing but just started 3 months ago.
That being said, if you are not yet in a situation as complex as the one your tool is designed to deal with, there is a very good chance you will waste some time starting to use such a tool "early." You might get that time back later when you scale, you might have the right people to set up the complex tool the right way for your simple situation, but you are taking a bit of a risk. As long as you go into the situation with your eyes open I think most people end up ok. The horror stories almost always come from people who are working to fulfill needs they do not have and don't understand why their work isn't giving good ROI.