Live data from Hacker News

Jepsen: Redpanda 21.10.1

jepsen.io

51–60 of 61 posts

Re: Jepsen: Redpanda 21.10.1

#51

This isn't anything against Redpanda, but I'm always amazed how badly all these distributed databases do in Jepsen. What would one use them for in practice, which wouldn't be better suitable by a (the thing I've used), say postgresql and streaming replication in case the server goes down? (I'm not suggesting there isn't a good application, just I'm not knowledgeable enough to know of one).

CAP-theorem and configuration. When choosing a distributed database you are in a way already by definition giving up a chunk of of the C, Consistency.

Once you have a distributed database, they often have a myriad of tuning parameters that all impact in which corner of the CAP triangle you want to be. Using this you have to choose what risks you are willing to accept. If all my replicas are all in the same rack, any timeout-issues found would often just be academical and I can pragmatically design such a system very differently, vs if they are in different parts of the globe.

I might also have such high influx of low value data that I can accept some losses, the cost of a total deadlock would be more than just one transaction lost in cyberspace, not everyone is building a bank. That said, it still sucks a lot to have inconsistent data regardless of your application, so in such cases aim for lost data rather than wrong data. So in theory, the distributed approach is consider wrong, but in practice it might just be good enough.

This also ties into how you model your data, a lot of the faults found in the latest mongodb analysis was around multi-document transactions, but nobody uses mongodb this way, mostly it’s just a place where you dump standalone documents into it.

In the end, you have to go back to original question of why you are choosing a distributed DB in the first place, is it for scale, HA, regionality or other reasons. Then design around that, it’s never a silver bullet fix all solution.

Taking your example of streaming replication. How would that behave if primary acked to the client and then crashed, before the replica received the transaction? The alternative of waiting for the replica before you ack instead gives you 2 sources of failure. You’ve now reinvented a distributed system and are now in the same soup as all these other databases, just with other tradeoffs. :)

Re: Jepsen: Redpanda 21.10.1

#52
post #35

This report seems to have some wrong insights. Auto-commit offsets doesn't imply dataloss if records are processed synchronously. This is the safest way to test Kafka instead of commit offsets manually

Can you clarify what you mean? AFAIK with manual commit you have the most control over when the commit happens Look at this blog post describing a data loss caused by auto-commit: https://newrelic.com/blog/best-practices/kafka-consumer-conf... Also there also may be more subtle issues with auto-commit: https://github.com/edenhill/librdkafka/issues/2782

I'm afraid the article is also wrong, this is a typical misconception when working with Kafka. Offsets are committed in the next poll() invocation. If the previous messages weren't processed, a rebalance occurs and messages are processed by other instance. This is an implementation detail of the Java client library but it allows the at-least-once semantic with auto-commit. The book Effective Kafka has a better explanation.

librdkafka isn't part of official Kafka so it may have problems with this as it has other limitations.

In any case, the report isn't right about this and it doesn't use the safest options. Commit offsets manually is the most flexible way but it isn't easy, being the error more usual to commit offsets individually

Re: Jepsen: Redpanda 21.10.1

#53
post #16
post #14

Earlier quoted context omitted.

Are there good research groups or journals to follow to keep apprised of the state of the art here?

I've created this list a while ago https://github.com/redpanda-data/awesome-distributed-transac... . Maybe it's time to update it. Usually I start with a couple of seed papers then follow the references, look at the other papers the authors wrote. When a phd student explores an area they write several paper on the topic so there is a lot material to read. But the real gem is the thesis, it has depth, context and a lo…

Thank you for putting this together!

Re: Jepsen: Redpanda 21.10.1

#54
post #41

Earlier quoted context omitted.

I happened to know RedPanda founder back in the days he was at Concord.io (as a founder and a main dev). The level of obsession with performance and optimization of this guy was insane. He's not only extremelly skilled with C++, but also very passionate about rethinking large and complex systems and rebuilding them to enable 10-100x speed improvements. It's like his personal hobby – take a piece of software everyone…

Does he give conference talks that I might find on YouTube?

Ha, good question. There are some:

Alex & Kelsey Hightower talk for intro: - https://www.youtube.com/watch?v=EXpkRR2sJbc

Some talks (haven't watched them, super geeky):

[1] https://www.youtube.com/watch?v=JPTt1ny67x4

[2] https://www.youtube.com/watch?v=xyN4vrLGjxo&

[3] https://www.youtube.com/watch?v=eFOZx7dP9sc

[4] https://www.youtube.com/watch?v=UXJhmWnSzc0

Re: Jepsen: Redpanda 21.10.1

#55

The level of intellectual discipline and competence on display here is inspiring. I'd love to take one of the Jepsen courses, but it seems they're offered only as corporate training. Maybe my employeer will agree to bring them in. For now I'll have to satisfy myself with the YouTube videos.

There's been a lot of interest in this, and I keep meaning to put together an open-to-the-public class when I'm less swamped! Might get a chance to do this shortly--I'll post on https://groups.google.com/a/jepsen.io/g/announce when it happens.

Re: Jepsen: Redpanda 21.10.1

#56
post #51

This isn't anything against Redpanda, but I'm always amazed how badly all these distributed databases do in Jepsen. What would one use them for in practice, which wouldn't be better suitable by a (the thing I've used), say postgresql and streaming replication in case the server goes down? (I'm not suggesting there isn't a good application, just I'm not knowledgeable enough to know of one).

CAP-theorem and configuration. When choosing a distributed database you are in a way already by definition giving up a chunk of of the C, Consistency. Once you have a distributed database, they often have a myriad of tuning parameters that all impact in which corner of the CAP triangle you want to be. Using this you have to choose what risks you are willing to accept. If all my replicas are all in the same rack, any…

> When choosing a distributed database you are in a way already by definition giving up a chunk of of the C, Consistency.

Er, I don't think that's right. Distributed systems (even those running in asynchronous networks) can and often do satisfy linearizability. Gilbert & Lynch's proof of the CAP theorem just says that if you do choose linearizability (C) in an asynchronous network (P), you can't also guarantee total availabilility (A)--under some network faults, some operations may not complete. https://users.ece.cmu.edu/~adrian/731-sp04/readings/GL-cap.p...

Re: Jepsen: Redpanda 21.10.1

#57
post #35

Earlier quoted context omitted.

Can you clarify what you mean? AFAIK with manual commit you have the most control over when the commit happens Look at this blog post describing a data loss caused by auto-commit: https://newrelic.com/blog/best-practices/kafka-consumer-conf... Also there also may be more subtle issues with auto-commit: https://github.com/edenhill/librdkafka/issues/2782

I'm afraid the article is also wrong, this is a typical misconception when working with Kafka. Offsets are committed in the next poll() invocation. If the previous messages weren't processed, a rebalance occurs and messages are processed by other instance. This is an implementation detail of the Java client library but it allows the at-least-once semantic with auto-commit. The book Effective Kafka has a better explan…

> Offsets are committed in the next poll() invocation.

I'm a little surprised by this--not that you're necessarily wrong, but our tests consumed messages synchronously, and IIRC (pardon, it's been 3 months since I was working on Redpanda full time and my time to go get a repro case is a bit limited) did see lost messages with the default autocommit behavior. At some point I'll have to go dig into this again.

Re: Jepsen: Redpanda 21.10.1

#58
post #12

The first half is jepsen team trying to divine some actual testable guarantees from a pile of blog posts and a random Google doc. What a mess.

The mess is mostly the result of the mismatch between the classic database transactional model and kafka transactional model (G0 anomaly). If you read the documentation without the database background it seems ok, but when you notice the differences between the models it becomes hard to understand if it's a bug or property of the Kafka protocol. There is a lot of research happening around this area even in the databa…

I agree--rystsov has covered this well here and in other parts of the thread. I just want to add that some of the Kafka documentation did claim writes were isolated (where other Kafka documentation contradicts that claim!) so it's possible that depending on which parts of the docs users read, they might expect that G0 was prohibited. That's why this report discusses it in such detail. :-)

Re: Jepsen: Redpanda 21.10.1

#59

This report seems to have some wrong insights. Auto-commit offsets doesn't imply dataloss if records are processed synchronously. This is the safest way to test Kafka instead of commit offsets manually

Almost nobody auto commits offsets in real applications though. If you do then you should really stop :)
Post reply on HN