Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

161–170 of 295 posts

Re: Kafka is dead, long live Kafka

#161

Post by diarrhea »

slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

Ironically, I have a fundamental hatred for LinkedIn and its sluggishness. It’s one of the slowest websites I frequent. I have few connections all things considered. Putting my feed together cannot be rocket science (its contents stay quite static for sometimes weeks at a time).

Re: Kafka is dead, long live Kafka

#162

Post by richieartoul »

pshirshov wrote: post #142
richieartoul wrote: post #45

Earlier quoted context omitted.

(WarpStream co-founder) We use S3 for data replication, availability, and durability, but not for the log ordering guarantees. For that, we have our own custom metadata store that "imposes" ordering guarantees on top of the data that is written to S3. We don't rely on any kind of timestamp-based last write wins policy for concurrent writes. In fact the agents will never write a file to S3 with the same name more than…

So, in fact it's not really stateless, you have your locking/ordering layer and you implement some consensus for that layer?

Its stateless from the customer's perspective since they don't have to manage the consensus layer (which is only possible because we separated data from metadata).

Obviously there is state somewhere since the cluster is storing data, its just offloaded to S3 + our control plane, while still keeping all the data in the customer's cloud account.

Re: Kafka is dead, long live Kafka

#163

Post by EdwardDiego »

tofuahdude wrote: post #159
EdwardDiego wrote: post #150

Earlier quoted context omitted.

It's really good when you're producing a lot of data fast that you don't want to lose, and you want multiple consumers to read. It's a complex tool that solves a complicated problem. But if you don't actually have that problem, then that's a whole lot of complexity for no gain.

Hypothetical use case I've been thinking about: Say I want to log all http requests to the server (I know I said a keyword of log) and then process those logs into aggregates, stick them in a time series. Would it be insane to "log" everything into kafka? Or what would be the more "correct" tool for that job?

No, that's a common use case. Ultimately Kafka is just a big dumb pipe. Only question I'd ask is how much data are you expecting?

Re: Kafka is dead, long live Kafka

#164

Post by relay23 »

snapetom wrote: post #106
carlio wrote: post #78

Earlier quoted context omitted.

I work as a contractor so I move between places. I have found a few companies trying to introduce kafka, and every time it has been a solution in search of a problem. I don't doubt that it has a good use case but I have so far only encountered the zealots who crowbar it into any situation and that has left a residual bad taste in my mouth. So I fall into the "hate it" side.

> and every time it has been a solution in search of a problem. More refined to this, in my experience at the last two jobs, the queue problem is there, but the Kafka solution is based solely on "enterpriseyness" of Kafka, not any practical reason. RabbitMQ is highly performant, SQS is really easy. Both are great queues. Kafka is muuch more, yet, Kafka is chosen because "it's enterprise."

Maybe "it's enterprise" means that's what the enterprise standardized on. There are a couple of practical reasons that come to mind on why that's the case - a) it's more resilient and durable than messaging platforms, and b) it is a platform of dumb pipes, so to make it a central data bus managed by platform teams means that they don't have to get into the detail of which queues perform what functions, have what characteristics, etc. Rather the client teams in the various business units can take care of all of their "smarts" the way they want. It also handles log/telemetry ingestion, data platform integration, and interservice comms use cases which is pretty multi-functional. That's the primary reason why Kafka has become such a pervasive and common platform, it's not because it's trendy, in fact most operations teams would rather not even have to operate the kafka platform.

Re: Kafka is dead, long live Kafka

#166

Post by majormajor »

parhamn wrote: post #146
benjaminwootton wrote: post #128

Earlier quoted context omitted.

Why? Its quite easy to use Kafka as a messaging queue without even thinking about the write ahead log semantics. It’s there if you need it, but Kafka scales down to being a message broker fairly well in my opinion.

Because operationalizing Kafka is difficult from a infrastructure (scala/java, zookeper, durable disk management, lots of moving parts), learning and a code perspective (pointer tracking, partition management, delegation, etc) relative to the other pubsub/mq tools. So if you don't have it operationalized and your use case is simple, it makes most sense to use a simpler tool (rmq/ampq, cloud pubsub, nsq, etc, perhaps…

If I could get away with a vendor cloud queue I wouldn't move to Kafka for the hell of it, but if I needed higher volume data shipping I've never found the infra as hard it people make it out to be. Unless you're doing insane volumes in single clusters, most of the pieces around it can work OK on default mode for a surprisingly long time.

You can cost footgun yourself like the blog here talks about with cross-AZ stuff (but that doesn't feel like the right level to do that at for me for most cases anyway), and anytime you're doing events or streaming data at all you're gonna run into some really interesting semantic problems compared to traditional services (but also new capacities that are rarely even attempted in that world, like replaying failed messages from hours ago), so it's good to know exactly what you're getting into, but I've spent far less time fighting ZK than Kafka and far less time fighting either than getting the application semantics right.

I imagine a lot of pain comes from "I want events, I know nothing about events, I don't know how to select a tool, now I'm learning both the tool and the semantics of events and queues both on the fly and making painful decisions along the way" which I've seen several places (and helped avoid in some of the later places after learning some hard, not-well-discussed-online lessons). I think the space just lets you do so many more things, so figuring out what's best for YOU is way more difficult the first time you as traditional-backend-online-service-developer start asking questions like "but what if we reprocess the stuff that we otherwise would've just black-hole-500'd during that outage after all" and then have to deal with things like ordering and time in all its glory.

Re: Kafka is dead, long live Kafka

#167

Post by cgio »

ryanworl wrote: post #124
poorman wrote: post #121

Earlier quoted context omitted.

It sounds like there's a sweet spot here. If you are not ACKing Produce requests for 100ms then there's a huge amount latency. If the user want's to reduce that latency from 100ms to say 1ms then their S3 GET requests cost just went up by 100x.

[WarpStream co-founder here] We've done lots of customer research here and, combined with the experience my co-founder and I have, we can confidently say most Kafka users (especially high-throughput users) would happily make a trade off of increased end-to-end latency in exchange for a massive cost reduction and the operational simplicity provided by WarpStream.

I don’t want to disagree with the research here, but what is not evident from the article is that this is not a magical solution that improves upon Kafka hands down, but rather a solution that addresses trade offs someone might be willing to entertain. I think on the query side things may be quite suboptimal in this setup if I understand it correctly. Correct me if I am wrong but if two agents write on a single topic, I would need to read two files to consume. Also I remember infamous stories about the cost of deleting data from S3, how do you tackle that if you have that many individual files? With these trade offs how does the solution compare to using Aurora?

Re: Kafka is dead, long live Kafka

#168

Post by whalesalad »

whatscooking wrote: post #52
whalesalad wrote: post #28

Running Kafka is an exercise in sadomasochism. This on the other hand sounds pretty neat.

What? It’s pretty easy to set up and get going

The last time I was running a cluster it was 2014-2015 and it was not too bad to get up and running, but it felt so heavy handed for what it was. The zookeeper requirement was frustrating. The fact that consumers needed to maintain offset state was frustrating (although... understandable). For the right use case I suppose it is 100% worthwhile but more often than not I have seen it used in places where other tools would be more appropriate.

Re: Kafka is dead, long live Kafka

#169

Post by parhamn »

morelisp wrote: post #151
benjaminwootton wrote: post #128

Earlier quoted context omitted.

Why? Its quite easy to use Kafka as a messaging queue without even thinking about the write ahead log semantics. It’s there if you need it, but Kafka scales down to being a message broker fairly well in my opinion.

Besides the operational concerns mentioned in the sibling comment, Kafka is simply not a great queue. You can't work-steal, you can't easily retry out-of-order, you can't size retention based on "is it processed yet", and you may need to manually implement DLQ behavior. If you already have Kafka for other (more WAL-y, or maybe older log-shippy) reasons it can be an OK queue, especially if you've got a team that can u…

Great point. The basic semantics are very different too. In MQs you partition/namespace/channel (whatever you want to call it) based on how data flows in your application (e.g. fanout). In Kafka you're tied more to a persistance model so you end up with fat linear topics and the "filtering"/flow management happen on the consumer's side.

Re: Kafka is dead, long live Kafka

#170

Post by richieartoul »

ryanworl wrote: post #124
poorman wrote: post #121

Earlier quoted context omitted.

It sounds like there's a sweet spot here. If you are not ACKing Produce requests for 100ms then there's a huge amount latency. If the user want's to reduce that latency from 100ms to say 1ms then their S3 GET requests cost just went up by 100x.

[WarpStream co-founder here] We've done lots of customer research here and, combined with the experience my co-founder and I have, we can confidently say most Kafka users (especially high-throughput users) would happily make a trade off of increased end-to-end latency in exchange for a massive cost reduction and the operational simplicity provided by WarpStream.

Replying here instead of below because we hit depth limit. WarpStream definitely isn’t magical, it makes a very real trade off around latency.

On the read side, the architecture is such that you’ll have to pay for 1 GET request for every 4 MiB of data produced for each availability zone you run in. If you do the math on this, it is much cheaper than manually replicating data across zones and paying for interzone networking.

RE:deletes. Deleting files in S3 is free, it can just be a bit annoying to do but the WarpStream agents manage that automatically. It’s creating files that is expensive, but the WarpStream storage engine is designed to minimize this.

I will do a future blog post on how we keep S3 GET costs minimal, it’s difficult to explain in a HN comment on mobile. Feel free to shoot us an email at founders@warpstreamlabs.com or join our slack if you care for a more in depth explanation later!

Post reply on HN