Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

291–295 of 295 posts

Re: Kafka is dead, long live Kafka

#291

Post by valenterry »

regularfry wrote: post #290
valenterry wrote: post #289

Earlier quoted context omitted.

You are now assuming that they indeed have a database nd store users in there, and not use a 3rd party system that might not even allow to attach meta data (or at least make it difficult).

In which case why is the password reset my problem at all? But even if I've handed off identity management entirely, I almost certainly do have some per-user state. Otherwise... what on earth am I doing with users in the first place?

Well, maybe because the 3rd party service is not reliable or you want to send an additional mail.

But okay, fair point - password-reset is maybe not the greatest example. But it doesn't invalidate the general point (I gave a couple of other examples).

> Otherwise... what on earth am I doing with users in the first place?

Maybe just making sure that the user is known and has paid for plan X (if the 3rd party service offers that).

Re: Kafka is dead, long live Kafka

#292

Post by bit_flipper »

AtlasBarfed wrote: post #284
bit_flipper wrote: post #278

Earlier quoted context omitted.

Not your main point, but MongoDB didn't commission Kyle to do that report as they had in the past, he did it on his own time. That's why his report doesn't mention repeat testing. They do actually run his tests in their CI and those new tests were used to isolate that specific bug. Moreover, some of the complaints about weak durability defaults for writing were later fixed: https://www.mongodb.com/blog/post/default-m…

Do you have a source for this? I got the impression at the time that there was some commissioning of his services, but that they didn't like the report. But he publishes work, and released the report, which forced them to deal with it. Every distributed tech fails when he test it, but the tenor and nature of the report for MongoDB was different. It basically said between the lines "do not use this product". MongoDB h…

Source for which claim? Kyle was paid for work testing 3.4.0-rc3[1] and 3.6.4[2] which analyzed single document concurrency in a sharded configuration. Those tests run in their CI [3]. MongoDB had some somewhat misleading copy on their website about the result of those tests, so Kyle decided to test the new multi-document transactions feature for 4.2.6 and found some bugs.

It's fair to not trust the database or company, I don't blame you for that. But I think Kyle's MongoDB 4.2.6 report was not nearly as concerning as his PostgreSQL 12.3 report which found serializability bugs in a single instance configuration, among other surprising behaviors. MongoDB's bugs were at least in a new feature in a sharded configuration. I don't think his most recent report was actually as negative as it may read to you. I say this as someone who mostly runs PostgreSQL, by the way!

As a side note I believe there are consistency bugs existing right now in both MongoDB and PostgreSQL (and MySQL and Cassandra and Cockroachdb and...) waiting to be discovered. I'm a jaded distributed systems operator :)

[1] https://jepsen.io/analyses/mongodb-3-4-0-rc3

[2] https://jepsen.io/analyses/mongodb-3-6-4

[3] https://github.com/search?q=repo%3Amongodb%2Fmongo+jepsen&ty... (note: not an expert in when or what suites it runs, just have seen it running before as a demo)

Re: Kafka is dead, long live Kafka

#294

Post by geekone »

ryanworl wrote: post #2

I'm Ryan Worl, co-founder and CTO of WarpStream. We're super excited to announce our Developer Preview of our Kafka protocol compatible streaming system built directly on top of S3 with no stateful disks/nodes to run, no rebalancing data, no ZooKeeper, and 5-10x cheaper because of no cross-AZ bandwidth charges. If you have any questions about WarpStream, my co-founder (richieartoul) and I will be here to answer them.

What was used to generate the diagrams in the post?

Re: Kafka is dead, long live Kafka

#295

Post by physicles »

mdaniel wrote: post #263
physicles wrote: post #240

Earlier quoted context omitted.

Not my experience at all. We’ve been running a 3 node cluster for several years, and a significant minority of the times I’ve been paged is because ZK got into a bad state that was fixed by a restart (what bad state exactly? Don’t know, don’t care, don’t have two spare weeks to spend figuring it out). Note that we have proper liveness checks on individual instances, so the issue is more complicated than that. Migrate…

> Migrated to 3.3 with KRaft about half a year ago Did you follow their migration guide, or did you just rebuild the cluster and then using KRaft? I didn't know how "migration" was used in that context

We built a new cluster and very carefully rolled over our producers and consumers. It wasn’t simple, but it’s possible to do without downtime.
Post reply on HN