Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why was Apache Kafka created?

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

141–150 of 229 posts

Re: Why was Apache Kafka created?

#141

Post by mrkeen »

AtlasBarfed wrote: post #59
zug_zug wrote: post #39

Earlier quoted context omitted.

Make it less event-orchestrated and use a db. It’s just a social network for recruiters it’s not as complicated as they like to pretend. You don’t need push, it’s just a performance optimization that almost never justifies using a whole new tool.

Your solution to a queue and publish subscribe problem is to use a database?

Adding onto this.

> LinkedIn used site activity data (e.g. someone liked this, someone posted this)1 for many things - tracking fraud/abuse, matching jobs to users, training ML models, basic features of the website (e.g who viewed your profile, the newsfeed), warehouse ingestion for offline analysis/reporting and etc.

Who controls the database? Is it the fraud/abuse team responsible for the migrations? Does the ML team tell the Newsfeed team to stop doing so many writes because it's slowing things down?

Re: Why was Apache Kafka created?

#142

Post by nchmy »

dijit wrote: post #68
jeremyjh wrote: post #64

Earlier quoted context omitted.

This doesn’t really support your position as far as most readers are concerned - it sounds like a disconnect. If they didn’t do this in any ad copy or public docs it’s not really in Mongo territory.

I don’t really care. I’m telling you why I am skeptical of any tech that intentionally obfuscates trade-offs, I’m not making a comparison on which of these is worse; and I don’t really care if people take my anecdote seriously either : because they should make their own conclusions. However it might help people go in to a topic about performance and reliability from a more informed position.

I don't doubt your experience. But I think it might have been more just that guy, than NATS in general.

The other day i was listening to a podcast with their ceo from maybe 6 months ago, and he talked quite openly about how jetstream and consumers add considerable drag compared to normal pubsub. And, more generally, how users unexpectedly use and abuse nats, and how they've been able to improve things as a result.

Re: Why was Apache Kafka created?

#143

Post by HarHarVeryFunny »

Joeri wrote: post #37
mrweasel wrote: post #24

Earlier quoted context omitted.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

Kafka isn’t a queue, it’s a distributed log. A partitioned topic can take very large volumes of message writes, persist them indefinitely, deliver them to any subscriber in-order and at-least-once (even for subscribers added after the message was published), and do all of that distributed and HA. If you need all those things, there just are not a lot of options.

Why do you say log rather than just publish and subscribe?

Re: Why was Apache Kafka created?

#144

Post by exitb »

HarHarVeryFunny wrote: post #143
Joeri wrote: post #37

Earlier quoted context omitted.

Kafka isn’t a queue, it’s a distributed log. A partitioned topic can take very large volumes of message writes, persist them indefinitely, deliver them to any subscriber in-order and at-least-once (even for subscribers added after the message was published), and do all of that distributed and HA. If you need all those things, there just are not a lot of options.

Why do you say log rather than just publish and subscribe?

Clients don’t have to subscribe to latest messages, but rather can request any available offset range.

Re: Why was Apache Kafka created?

#145

Post by HarHarVeryFunny »

mrweasel wrote: post #24
kachapopopow wrote: post #5

As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

IMO, recommending RabbitMQ depends on what language you are using and how well suited the available client libraries are to your use case.

I used RabbitMQ a few years back on a C++ project, and at the time (has anything changed?) the best supported C++ client library seemed to be AMQP-CPP which isn't multi-thread safe, therefore requiring an application interface layer to need to be written to address this in a performant way.

In our case we wanted to migrate a large legacy system from CORBA (point to point) to a more flexible bus-based architecture, so we also had to implement a CORBA-like RPC layer on top of Rabbit, including support for synchronous delivery failure detection, which required more infrastructure to be built on top of AMQP-CPP. In the end the migration was successful, but it felt like we were fighting AMQP-CPP a lot of the way.

Re: Why was Apache Kafka created?

#146

Post by 0x445442 »

SJC_Hacker wrote: post #109
AtlasBarfed wrote: post #50

Earlier quoted context omitted.

If you're running a distributed system... You're running a distributed system. They aren't simple. Especially on AWS. AWS is really a double-bladed sword. Yeah, you'll get tutorials to set up whatever distributed system pretty quickly, but your nodes aren't nearly as reliable. Your networking isn't nearly as reliable. Your costs aren't nearly as reliable and administration. Headaches go up in the long run

This is what alot of people don't get. They think Kubernetes is a solution to everything. In reality all k8s is makes the mechanics a little easier. But if you don't understand distributed systems, it almost makes it worse because its tempting to segment the system across dozens of microservices which all have to talk with each other and synchronize, and the whole thing becomes a buggy, slow clusterfuck.

Dozens of microservices, oh one should be so lucky. Try hundreds where the complexity of the whole system rises non-linerarly as a function of each microservice.

Re: Why was Apache Kafka created?

#147

Post by mannycalavera42 »

Joeri wrote: post #37
mrweasel wrote: post #24

Earlier quoted context omitted.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

Kafka isn’t a queue, it’s a distributed log. A partitioned topic can take very large volumes of message writes, persist them indefinitely, deliver them to any subscriber in-order and at-least-once (even for subscribers added after the message was published), and do all of that distributed and HA. If you need all those things, there just are not a lot of options.

HERO

Re: Why was Apache Kafka created?

#148

Post by nova22033 »

poemxo wrote: post #21
njitbew wrote: post #13

Earlier quoted context omitted.

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

Starting up a Java program takes much longer than it should and that affects perception.

It may affect developer perception but I'm pretty sure my users don't notice and don't care.

Re: Why was Apache Kafka created?

#149

Post by game_the0ry »

This might be hyperbolic, but I think Kafka (or at least the concept of event driven architecture for sharing data across many systems) is one of the most under-rated technologies. It's used at a lot of big corps but is never talked about.

Re: Why was Apache Kafka created?

#150

Post by enether »

erulabs wrote: post #67

Kafka's ability to ingest the firehose and present it as a throttle-able consumable to many different applications is great. If you're thinking "just use a database", it's worth noting that SQL databases are _not well suited_ to drinking from a firehose of writes, and that distributed SQL in 2012 was not a thing. Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was…

Love the story!

> Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was a big step forward for web applications at scale.

Could you expand on this - when does it drop C? Are you referring to cases where you write to Kafka without waiting for all replicas to acknowledge the write? (acks=1)

And why was it a big step - what other systems didn't embrace dropping the C?

Post reply on HN