This report seems to have some wrong insights. Auto-commit offsets doesn't imply dataloss if records are processed synchronously. This is the safest way to test Kafka instead of commit offsets manually
Jepsen: Redpanda 21.10.1
31–40 of 61 posts
Re: Jepsen: Redpanda 21.10.1
#32Redpanda (back when they were VectorizedIO) spammed my work email after I starred one of their repos, denied it after I called them out on it and I just noticed that they had deleted their response to me. Pretty sneaky to go back and delete the tweets first denying and then apologizing. Receipts: https://twitter.com/d11cc3s/status/1447573471152656389 https://twitter.com/d11cc3s/status/1450906855115354116
Sounds like you have a personal, singular issue with them that I can't imagine anyone else cares about.
Re: Jepsen: Redpanda 21.10.1
#33This isn't anything against Redpanda, but I'm always amazed how badly all these distributed databases do in Jepsen. What would one use them for in practice, which wouldn't be better suitable by a (the thing I've used), say postgresql and streaming replication in case the server goes down? (I'm not suggesting there isn't a good application, just I'm not knowledgeable enough to know of one).
I'm constantly surprised more folks don't use FoundationDB, I'm pretty sure the Jepsen folks said something to the tune of the way FoundationDB is tested is far beyond what Jepsen does (Good talk on FDB testing: https://www.youtube.com/watch?v=4fFDFbi3toc ). My read is that most use cases just need something that works _enough_ at scale that the product doesn't fall over and any issues introduced by such bugs can be…
Having looked at FoundationDB a bit it wasn't clear why I would choose it. It has transactions, which is nice, but not that big of a deal despite how much time they put into talking about it. I actually don't even need transactions since all of my writes commute, so it's particularly uninteresting to me.
They say they're fast, but I didn't find a ton of information about that.
Ultimately the sell seemed to be "nosql with transactions" and I just couldn't justify putting more time into it. I did watch their excellent talk on testing, and I respect that they've put that level of effort into it, and it was why I even considered it, but yeah, what am I missing?
Re: Jepsen: Redpanda 21.10.1
#34> A KafkaConsumer, by contrast, will happily connect to a jar of applesauce14 and return successful, empty result sets for every call to consumer.poll. This makes it surprisingly difficult to tell the difference between “everything is fine and I’m up to date” versus “the cluster is on fire”, and led to significant confusion in our tests.
This tickled my funny bone. Never expected humor in a Jepsen writeup. Kudos!
Re: Jepsen: Redpanda 21.10.1
#35This report seems to have some wrong insights. Auto-commit offsets doesn't imply dataloss if records are processed synchronously. This is the safest way to test Kafka instead of commit offsets manually
Can you clarify what you mean? AFAIK with manual commit you have the most control over when the commit happens
Look at this blog post describing a data loss caused by auto-commit: https://newrelic.com/blog/best-practices/kafka-consumer-conf...
Also there also may be more subtle issues with auto-commit: https://github.com/edenhill/librdkafka/issues/2782
Re: Jepsen: Redpanda 21.10.1
#36> A KafkaConsumer, by contrast, will happily connect to a jar of applesauce14 and return successful, empty result sets for every call to consumer.poll. This makes it surprisingly difficult to tell the difference between “everything is fine and I’m up to date” versus “the cluster is on fire”, and led to significant confusion in our tests. This tickled my funny bone. Never expected humor in a Jepsen writeup. Kudos!
> Never expected humor in a Jepsen writeup
Jepsen reports are often pretty funny, some famously so
Re: Jepsen: Redpanda 21.10.1
#37> A KafkaConsumer, by contrast, will happily connect to a jar of applesauce14 and return successful, empty result sets for every call to consumer.poll. This makes it surprisingly difficult to tell the difference between “everything is fine and I’m up to date” versus “the cluster is on fire”, and led to significant confusion in our tests. This tickled my funny bone. Never expected humor in a Jepsen writeup. Kudos!
Wait until you find out why it's called “Jepson”
Re: Jepsen: Redpanda 21.10.1
#38Always great to read this. I preformed a jenkins test on Microsoft internal infra and it's a huge insight. From an academic side it's just as interesting looking into the lack of standards within consistently and the definitions of them.
Cool! What did you test? I've played with Jepsen and Cosmos DB when I was at Microsoft but we had to ditch ssh, write custom agent and inject faults with PowerShell command lets.
Re: Jepsen: Redpanda 21.10.1
#39This isn't anything against Redpanda, but I'm always amazed how badly all these distributed databases do in Jepsen. What would one use them for in practice, which wouldn't be better suitable by a (the thing I've used), say postgresql and streaming replication in case the server goes down? (I'm not suggesting there isn't a good application, just I'm not knowledgeable enough to know of one).
I'm constantly surprised more folks don't use FoundationDB, I'm pretty sure the Jepsen folks said something to the tune of the way FoundationDB is tested is far beyond what Jepsen does (Good talk on FDB testing: https://www.youtube.com/watch?v=4fFDFbi3toc ). My read is that most use cases just need something that works _enough_ at scale that the product doesn't fall over and any issues introduced by such bugs can be…
As someone who is switching to FoundationDB: because it's not easy. It doesn't look like other databases, it isn't in fashion (yes, these things matter), and it requires thinking and adapting your application to really use it to its full potential. It could also benefit from a bit more developer marketing.
But it's the best thing out there.
Re: Jepsen: Redpanda 21.10.1
#40Hey folks, I was working with Kyle Kingsbury on this report from the Redpanda side and I'm happy to help if you have questions
A complete nit, but the testimonial from the CTO of The Seventh Sense on https://redpanda.com/ spells Redpanda as "Redpand".