Live data from Hacker News

MongoDB 3.4.0-rc3

jepsen.io

111–120 of 165 posts

Re: MongoDB 3.4.0-rc3

#111

Earlier quoted context omitted.

Cassandra testing on Jepsen. Interesting definition of "fare far better in Jepsen" https://aphyr.com/posts/294-jepsen-cassandra

Cassandra had a minor bug with timestamps which was fixed relatively fast and now it passes Jepsen fully. There were also a few implementation issues around LWT (Paxos) but at that time Paxos was very new. There were no architectural / fundamental changes needed to correct these issues. Also, even before fixing, Cassandra was losing only a tiny fraction of data than Mongo was losing. So, yes, this is what I call "far…

Cassandra had a minor bug with timestamps which was fixed relatively fast and now it passes Jepsen fully.

Wait, what?

Re: MongoDB 3.4.0-rc3

#112
post #20

Earlier quoted context omitted.

While you are correct that 3.4 now passes, I feel like your interpretation here is a bit optimistic. It's not like I tested Mongo and it passed out of the box--it failed Jepsen tests, and not just with a read anomaly--it lost majority-acknowledged inserts. The v0 protocol still fails--it's fundamentally broken. The v1 bugs are fixed now, but that's a consequence of our collaboration. I'd say the marker of maturity he…

I'd like to use the occasion to thank you for the service you provide to the community. The tests themselves, the collaboration with vendors, the competition it fosters and the in-depth technical write-ups I think push the whole ecosystem forward, which everyone eventually benefits from. Much obliged.

I agree -- I think the two most important people currently in software are aphyr for database testing, and lcamtuf for fuzzing (AFL). Both are pushing important fundamental software to pass the kind of tests it's embarrassing we haven't been doing for the past 20 years. I mean that in the most positive way, it's hard to get people to do things they've been neglecting for so long!

Re: MongoDB 3.4.0-rc3

#113
post #26

Earlier quoted context omitted.

I think maybe the doubt is based on case like https://aphyr.com/posts/282-jepsen-postgres which all systems are subject to.

Uh, no. That article is a bit of a long winded way to say that 2PC with timeouts is 2PC with timeouts.

It might be better to think of it as a limitation of two-generals, rather than 2PC in particular.

Re: MongoDB 3.4.0-rc3

#114
post #35
post #21

Earlier quoted context omitted.

So a client can get the analysis done and leave it unpublished if they don't like the results? Maybe that's standard practice but it doesn't seem great for users.

I agree. In my ideal world I publish everything immediately. There's no "standard practice" that I know of--very few people are doing this kind of work. Behind every one of these analyses is weeks of contract negotiation where I try to convince assorted lawyers & CFOs to go along with my weird, idealistic ethical policies. And conversely, those lawyers and CFOs do their best to balance the desire for correctness with…

Hey, just wanted to say I really appreciate all the in depth posts you put out on your site in hopes of educating others and helping them navigate the waters of distributed systems.

Curious, is there any single book you would say you've found to be the equivalent of say CLRS but for the fundamentals of distributed computing paradigms?

My current plan for deep diving into distributed systems theory has just been to go through resources like "Distributed systems for fun and profit", and aggregated lists/overviews like https://henryr.github.io/distributed-systems-readings/ and your distsys-class notes and then just DFS into certain topics from there - compared to the approach I've been taking with deep diving into OS and database theory which is just to go through text books like Operating System Concepts and Database System Concepts while applying the theory in side projects.

Re: MongoDB 3.4.0-rc3

#115
post #111

Earlier quoted context omitted.

Cassandra had a minor bug with timestamps which was fixed relatively fast and now it passes Jepsen fully. There were also a few implementation issues around LWT (Paxos) but at that time Paxos was very new. There were no architectural / fundamental changes needed to correct these issues. Also, even before fixing, Cassandra was losing only a tiny fraction of data than Mongo was losing. So, yes, this is what I call "far…

Cassandra had a minor bug with timestamps which was fixed relatively fast and now it passes Jepsen fully. Wait, what?

http://www.datastax.com/dev/blog/testing-apache-cassandra-wi...

Re: MongoDB 3.4.0-rc3

#116
I see many people say that Mongo is unfot for a production db, but it is actually used in production by many companies.

Obviously this is true of many products, but how serious is it in practice? The criteria for 'good enough' can't be passing Jespen because very few db's do.

Storing schemaless json is a very valid use case. And the fact is there aren't really many proper NoSql alternatives, other than RethinkDB, which I hope becomes popular, and maybe Couch. e.g. Cassandra is often touted but its a k-v store, not a document db.

There's a reason Mongo became and continues to be popular. I still think Rethink is superior in every way.

Re: MongoDB 3.4.0-rc3

#117

Earlier quoted context omitted.

>paid Kyle to help solve the problem. Any more information on this? How did he "solve" the problem? Edit: straight from the horse's mouth [0] [0] https://news.ycombinator.com/user?id=aphyr

Whoops, wrong link. Here's the correct one: https://news.ycombinator.com/item?id=13591048 It seems the problems were indeed "solved" and not solved .

I'm not exactly sure what you're trying to say, but perhaps I can help provide context:

In 2013, I performed an unpaid analysis, in my nights and weekends, of MongoDB. I found a bug leading to the loss of acknowledged writes with majority write concern. Mongo fixed this bug within a few weeks.

In 2015, I performed a followup test as a part of my work at Stripe. I confirmed dirty reads (which were already documented, though perhaps not well-appreciated), and discovered stale reads (which ran counter to MongoDB's documentation). MongoDB wasn't enthusiastic about that report initially, but got things sorted out and started work on adding majority and linearizable read concerns. I found that writes appeared linearizable.

In spring and summer of 2016, MongoDB paid me to help expand the Jepsen tests and hook them up to their internal CI system, so they could use the Jepsen tests to help verify their ongoing work towards linearizable reads. I privately confirmed that MongoDB still failed to prevent stale reads, but the dirty-read failures I'd seen in 2015 appeared to be prevented by majority reads.

In fall 2016, MongoDB announced they were almost ready to release 3.4.0 with support for linearizable reads, and paid me to perform a full analysis so they could be more confident in the results. It passed the linearizability tests I had initially written in 2015, but I offered to expand the tests to be more aggressive. Our collaboration resulted in the present analysis, uncovering design flaws in v0 and implementation bugs in v1. MongoDB worked to develop patches prior to 3.4.0's release, and that's why it passes now. :)

Re: MongoDB 3.4.0-rc3

#118

It's been a long way from the "Call Me Maybe: MongoDB" post from years back. Aphyr/Kyle took them to task in so many ways for playing fast and loose with data integrity, and rightly so. MongoDB could have said, "that guy's full of BS, ignore him," but instead they did the smart thing and paid Kyle to help solve the problem. n.b. I can't find the original "Call Me Maybe" post, but this later one [1] is similar. [1]: h…

>paid Kyle to help solve the problem. Any more information on this? How did he "solve" the problem? Edit: straight from the horse's mouth [0] [0] https://news.ycombinator.com/user?id=aphyr

I didn't say Kyle solved Mongo's problems, but that he helped solve the problems. He identified numerous data loss issues, explained them in great detail, and even has tooling to readily reproduce the data loss events. That's a massive contribution in my mind.

By comparison, it's the difference between a bug report that says "you lost my data, you suck!" and "I've identified [x] ways your product can lose data and here's exactly how to reproduce each one."

Re: MongoDB 3.4.0-rc3

#119
post #111

Earlier quoted context omitted.

Cassandra had a minor bug with timestamps which was fixed relatively fast and now it passes Jepsen fully. Wait, what?

http://www.datastax.com/dev/blog/testing-apache-cassandra-wi...

Pretty sure Cassandra still uses local timestamps and last-write-wins, which means you can't do safe updates. Also pretty sure they didn't fix the timestamp-collision row-isolation issue I talked about, because it was on HN's front page a few months ago. ;-)

Re: MongoDB 3.4.0-rc3

#120
post #108
post #85

Earlier quoted context omitted.

I don't think it should be taken as given that there's a correlation between competency and the size of an organization that a team exists within, and I don't think such a correlation, when combined with large organizations' usage of MongoDB would challenge the assertion that there exists an anticorrelation between team competency and use of MongoDB. Looking at the numbers, larger organizations straight-forwardly see…

I think you're looking at it wrong. It's not a popularity contest; I've seen billion dollar companies use fucking stupid tooling as well, but they still have the right processes where they don't lose data. Mongo, on the other hand, loses data.

In non-tech centric large orgs, it seems you frequently do not have the talent required to be be both risk averse and productive at the same time, so "IT" becomes a risk averse and non-productive political structure from which springs an "alternative IT" rebellion group (if the initiative is lead from above) or many little cowboy teams (if the initiative is driven from below) and these will be "productive" at the expense of having no processes for avoiding stupid risks that, amongst other things, can lead to data loss.
Post reply on HN