Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

61–70 of 295 posts

Re: Kafka is dead, long live Kafka

#61

Post by morelisp »

ryanworl wrote: post #29
morelisp wrote: post #24

Earlier quoted context omitted.

I see - this also explains the tiny consumer group / topic-partition limit of the free plan, then...

[WarpStream co-founder here] What do you think would be a good limit for the free plan? This isn't actually an architectural constraint for us. We just didn't want to promise unlimited usage forever so we picked a somewhat arbitrary number to start with.

I'm probably the wrong person to ask, sorry! My interest is purely academic, both for data governance and technical reasons I'll probably never be able to use this. I also don't mean to imply they're unfair, free is free - just that it explains why things which are pretty cheap in Kafka proper trigger billing (vs. of course things that are expensive in Kafka now being really cheap).

I do think a lot of people using Kafka compaction topics as their source-of-truth are defaulting to 256-~1000 partitions, if you want to entice them I'd probably offer at least 1024. (But I suspect your target market is not using these - rather they're salivating over the possibility of the automatic format transcoding you mentioned in another comment...)

Re: Kafka is dead, long live Kafka

#62

Post by insanitybit »

whatscooking wrote: post #52
whalesalad wrote: post #28

Running Kafka is an exercise in sadomasochism. This on the other hand sounds pretty neat.

What? It’s pretty easy to set up and get going

Getting a distributed set up plugged into your k8s/whatever framework can be a pain.

Re: Kafka is dead, long live Kafka

#63

Post by skybrian »

As a hobbyist I probably don’t need this but I’m curious. I haven’t used any of the S3-compatible data stores. Any ideas which would be easiest for messing around with?

Also, what databases are there built on S3? I think there is sqlite replication?

Re: Kafka is dead, long live Kafka

#64

Post by richieartoul »

88913527 wrote: post #60

It's refreshing to see cloud computing costs matter. Hopefully it'll cut down of needless cloud complexity and the proliferation of indirection in software abstractions of the last decade.

(WarpStream co-founder)

Cheers! We really want to drive the incremental cost (both in terms of $ and management overhead) per GiB down as low as possible so people can start using these systems for more use-cases.

Re: Kafka is dead, long live Kafka

#65

Post by bothra90 »

richieartoul wrote: post #15
dottedmag wrote: post #5

Earlier quoted context omitted.

Do you have a reference documentation for S3 data layout?

(WarpStream founder here) Not currently. One of the things we're looking to do next is make it so any topic can be "automatically" turned into a standard format in S3, something like Parquet/Iceberg/Deltalake so its easier to consume for application that don't particularly care about the Kafka protocol.

That would be awesome!

Re: Kafka is dead, long live Kafka

#67

Post by ryanworl »

nfw2 wrote: post #38

The title of the article should be "Kafka is dead. Long live Warpstream." The long live part refers to the successor.

(WarpStream CTO here) This is a bit subtle I will admit, but we view the Kafka protocol as a successor here because it will outlive Kafka the implementation.

Re: Kafka is dead, long live Kafka

#68

Post by slotrans »

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it.

(emphasis added)

Surely that's false?

Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it.

Linkedin built Kafka for massive-scale problems that 99% of us don't have. Though technologists have a well-earned reputation for using tech they don't need, my perception is that most of us are succeeding in avoiding the use of Kafka.

Re: Kafka is dead, long live Kafka

#69

Post by nithril »

rcarmo wrote: post #16

Well, one thing's for sure. Running Kafka on discrete VMs on a cloud provider "by the book" is ludicrously expensive. I remember having a very simple discussion with quite a few customers about both Kafka and Hadoop that boiled down to this: Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? (in this case it was Azure storage, which provides locally, redundant,…

> Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system?

Azure disk replications are for the durability of the data, not the avalability of the data from a kafka perspective.

Re: Kafka is dead, long live Kafka

#70

Post by richieartoul »

slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

(WarpStream co-founder)

That's fair, that statement is probably significantly colored by my own personal network / work experience.

Post reply on HN