Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

171–180 of 295 posts

Re: Kafka is dead, long live Kafka

#171

Post by _8j50 »

slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

A lot of companies use a product that uses kafka under the hoods. I was running graylog a few years ago for months before I knew kafka lay under the hoods.

They have a list: https://kafka.apache.org/powered-by

Re: Kafka is dead, long live Kafka

#173

Post by mtremsal »

richieartoul wrote: post #170
ryanworl wrote: post #124

Earlier quoted context omitted.

[WarpStream co-founder here] We've done lots of customer research here and, combined with the experience my co-founder and I have, we can confidently say most Kafka users (especially high-throughput users) would happily make a trade off of increased end-to-end latency in exchange for a massive cost reduction and the operational simplicity provided by WarpStream.

Replying here instead of below because we hit depth limit. WarpStream definitely isn’t magical, it makes a very real trade off around latency. On the read side, the architecture is such that you’ll have to pay for 1 GET request for every 4 MiB of data produced for each availability zone you run in. If you do the math on this, it is much cheaper than manually replicating data across zones and paying for interzone netw…

Very interesting trade-off! I was curious what you and Ryan were cooking post DDOG. "cost-effective serverless kafka" is a very interesting play. And congrats on the public announcement for "shipping Husky", finally. --Marc

Re: Kafka is dead, long live Kafka

#174

Post by yakkomajuri »

slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

I disagree. I certainly think it's possible people might be looking to fit Kafka into things that simply don't need it (perhaps driven by the system design theory focus in hiring), but for the applications where you have event streaming, Kafka is still the top choice. Analytics, messaging, sensors, etc.

From my side, I agree with the author about the "Accidental SRE" points. But Kafka is a solid technology, so much so that there's no shortage of "Kafka but better" tools out there (e.g. Redpanda).

Also you kind of drift off the point there at the end - even if it wasn't used extensively (a point of contention), that has nothing to do with whether it is polarizing or not? The statement about it being loved or hated is still relevant to those solving the 1% scaling problems you mentioned, even if 99% aren't.

It's like saying that the statement "lamborghinis are polarizing" is false because most of us don't have one? The author explicitly says "in the data space" too, effectively restricting the people he's talking about.

Re: Kafka is dead, long live Kafka

#175

Post by falcor84 »

parhamn wrote: post #88
slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

Kafka is a closer to a persistent WAL than a message queue. If your work doesn't need a WAL, it's almost certainly overkill and you will hate it. If your work needs a WAL then it'll be your favorite tool ever.

For those like me who aren't used to that abbreviation, it's short for Write-ahead Logging [0].

[0] https://en.wikipedia.org/wiki/Write-ahead_logging

Re: Kafka is dead, long live Kafka

#176

Post by agallego »

yakkomajuri wrote: post #174
slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

I disagree. I certainly think it's possible people might be looking to fit Kafka into things that simply don't need it (perhaps driven by the system design theory focus in hiring), but for the applications where you have event streaming, Kafka is still the top choice. Analytics, messaging, sensors, etc. From my side, I agree with the author about the "Accidental SRE" points. But Kafka is a solid technology, so much s…

that's a littlebit of a stretch. when you say "no shortage" - outside of redpanda what product exists that actually compete in all deployment modes?

it's a misconception that redpanda is simply a better kafka. the way to think about it is that is a new storage engine, from scratch, that speaks the kafka protocol. similar to all of the pgsql companies in a different space, i.e.: big table pgsql support is not a better postgres, fundamentally different tech. you can read the src and design here: https://github.com/redpanda-data/redpanda. or an electric car is not the same as a combustion engine, but only similar in that they are cars that take you from point a to point b.

Re: Kafka is dead, long live Kafka

#177

Post by pookah »

EdwardDiego wrote: post #152
snapetom wrote: post #106

Earlier quoted context omitted.

> and every time it has been a solution in search of a problem. More refined to this, in my experience at the last two jobs, the queue problem is there, but the Kafka solution is based solely on "enterpriseyness" of Kafka, not any practical reason. RabbitMQ is highly performant, SQS is really easy. Both are great queues. Kafka is muuch more, yet, Kafka is chosen because "it's enterprise."

Kafka isn't even a queue. I've done consulting on Kafka, and several times my recommendation is "You don't want or need Kafka". A classic sign of "you wanted an MQ" is when a consumer writes a message to a topic to let the producer know it read the message the producer wrote...

[dead]

Re: Kafka is dead, long live Kafka

#178

Post by pshirshov »

richieartoul wrote: post #162
pshirshov wrote: post #142

Earlier quoted context omitted.

So, in fact it's not really stateless, you have your locking/ordering layer and you implement some consensus for that layer?

Its stateless from the customer's perspective since they don't have to manage the consensus layer (which is only possible because we separated data from metadata). Obviously there is state somewhere since the cluster is storing data, its just offloaded to S3 + our control plane, while still keeping all the data in the customer's cloud account.

I think you should write some detailed docs about your design and consistency guarantees.

Re: Kafka is dead, long live Kafka

#179

Post by guru4consulting »

poorman wrote: post #121
ryanworl wrote: post #11

Earlier quoted context omitted.

[WarpStream co-founder and CTO here] 1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the number of Agents you run and the flushing interval, not the number of topic-partitions. We do not acknowledge Produce requests until data has been durably persisted in S3 and our clou…

It sounds like there's a sweet spot here. If you are not ACKing Produce requests for 100ms then there's a huge amount latency. If the user want's to reduce that latency from 100ms to say 1ms then their S3 GET requests cost just went up by 100x.

won't that be a problem for high-traffic topics? Kafka latency is usually in single digit milliseconds. For a topic with high throughput, a typical java client instance can send thousands of messages per second. When the acknowledgement latency increases to 1000ms, then the producer client would need to have multiple threads to handle the blocking calls. Either producer will have to scale to multiple instances, or else risk crashing with out-of-memory errors.

Re: Kafka is dead, long live Kafka

#180

Post by thephyber »

hughesjj wrote: post #122
whartung wrote: post #104

Earlier quoted context omitted.

There was recently an article about distributed systems that showed up here. (Harry Doyle: Christ, I can't find it. To hell with it!) And the author made a very interesting point about message queues. Simply, any problem that could be resolved by a message queue could be resolved by load balancing or persistence, and, therefore, messages queues were actually kind of a bad idea. There were two basic issues. The first…

Kafka is less message queue (rabbitmq, sqs) and more ordered stream/write ahead log (ala kinesis) > The first is that because of the nature of message queues, they're either empty or full. ... Wat? > that is it's a true "fire and forget" kind of request and direct response time is not a concern, then what does the queue gain you over simply posting it to a database table? Performance is why. The fire and forget aspec…

> > The first is that because of the nature of message queues, they're either empty or full.

> ... Wat?

I interpreted it as "they are either trending towards empty or full". The statement doesn't seem well thought through.

That might be true (either empty or full) most of the time (maybe) _if you squint_, but the entire point of the Message Queue is to provide buffering from the transient state (somewhere between empty and full) trending toward the empty state.

Post reply on HN