Live data from Hacker News

Call Me Maybe: MongoDB Stale Reads

aphyr.com

111–120 of 150 posts

Re: Call Me Maybe: MongoDB Stale Reads

#111
post #96

From 2009 to 2012 I had a distributed database startup that competed with MongoDB. We used Paxos for replication and built the database with on-disk consistency guarantees --- like the ones this article looks for and rightly obsesses over --- in mind. https://github.com/scalien/scaliendb Outcome: you've never heard of ScalienDB; MongoDB brilliantly won by winning the hearts and minds of hackers and coders who don't c…

I don't know, I got started with Mongo because ... it was so easy to start with, but come deployment time got bitten by a LOT of issues (which essentially negated any advantage in using Mongo).

As a result of this experience I almost exclusively use PostgreSQL, and I've never EVER been burned by taking this approach.

Sometimes I do use another DB but there has to be a seriously good reason for it.

Re: Call Me Maybe: MongoDB Stale Reads

#112
Since Postgres added a JSON type and Docker made running it simple in development, I haven't had a need for anything else. Call me old school, but I prefer starting with a relational database and changing when it's no longer appropriate.

Re: Call Me Maybe: MongoDB Stale Reads

#113
post #96

From 2009 to 2012 I had a distributed database startup that competed with MongoDB. We used Paxos for replication and built the database with on-disk consistency guarantees --- like the ones this article looks for and rightly obsesses over --- in mind. https://github.com/scalien/scaliendb Outcome: you've never heard of ScalienDB; MongoDB brilliantly won by winning the hearts and minds of hackers and coders who don't c…

we actually evaluated scaliendb extensively (even contributed patches) but decided against it because it was immature and notoriously instable.

Somewhat later, the open source repo scaliendb got pulled only to resurface years later. What happened?

Re: Call Me Maybe: MongoDB Stale Reads

#114

Earlier quoted context omitted.

HBase. With the caveats that it's piggy backing off the success of Zookeeper, and the tests were not run by Kyle himself: https://eng.yammer.com/call-me-maybe-hbase/

Note that Robert Yokota's addendum[1] points out that HBase "cannot achieve both consistency and availability." His earlier results did not take into account that HBase clients continuously retry failed ops. (According to Nicolas Liochon, upon its death a server's regions are moved to another node.) Failures start rolling in once network partition(s) extend beyond the configured timeout. Kyle [2] was not impressed: "…

No system can achieve 100% consistency and 100% availability in the presence of partitions. It's kind of wacky that Aphyr compared those replicated consensus tools and eventually consistent stores with HBase. HBase does not use a consensus protocol for replication, it uses HDFS. HBase is not eventually consistent. There is a single authoritative server for reads and writes of a lexicographic range of keys (a region) which writes immutable store files and a WAL to HDFS. Partial availability may be achievable for reads with a significant amount of effort and latency and limiting of total cluster size by allowing non-authoritative regionservers to read the HDFS WAL + Storefiles, but this really isn't realistic. I've personally been burned by the client retry thing though, and there's not really a better solution when you consider the types of workloads HBase is actually used for and it's incredibly variable latency (it aims only for consistency and very high AVERAGE throughput, at the cost of extremely high HIGHEST latency). One solution here could be the use of a configurable filesystem queue for clients. This is how you build resilient high-throughput pipelines. HBase is used in some places for OLTP, but only when the readload is very very low. So the effort to make reads more highly available would be in vain.

Re: Call Me Maybe: MongoDB Stale Reads

#115

Earlier quoted context omitted.

HBase. With the caveats that it's piggy backing off the success of Zookeeper, and the tests were not run by Kyle himself: https://eng.yammer.com/call-me-maybe-hbase/

Note that Robert Yokota's addendum[1] points out that HBase "cannot achieve both consistency and availability." His earlier results did not take into account that HBase clients continuously retry failed ops. (According to Nicolas Liochon, upon its death a server's regions are moved to another node.) Failures start rolling in once network partition(s) extend beyond the configured timeout. Kyle [2] was not impressed: "…

No database can "achieve both consistency and availability" during a partition.

Also, if you follow the rest of the twitter conversation you may realize, as they did, that only requests to the minority partition are unsuccessful - which is exactly what you want from CP.

Re: Call Me Maybe: MongoDB Stale Reads

#116
post #81

Earlier quoted context omitted.

I think at one point the employees of Basho knew how to write distributed DBs - Riak is the most advanced AP DB from a distributed systems theory perspective. However, in recent months, their CEO, CTO and Chief Architect have left, as well as many of their prominent engineers. Worryingly, the new CTO seems content to make inane comments about "Data Gravity" [0]. [0] - http://www.kdnuggets.com/2015/03/interview-dave-m…

I think Riak's theory is fine, but theory isn't enough. And they may have succumbed to the Osborne effect with Riak 2.0. Here's what I mean. Think of all the nice things you expect to come out of Riak's theoretical basis -- bulletproof distributed writes, for example. Well, the default last-write-wins writes aren't bulletproof. They clearly fail Jepsen [1]. And if you turned off last-write-wins, then you'd have to ha…

FWIW I did write a bunch of documentation on sibling resolution in Riak: http://docs.basho.com/riak/latest/dev/using/conflict-resolut...

Re: Call Me Maybe: MongoDB Stale Reads

#117
I must admit, I always feel like I am missing something in these discussions. Like I didn't get some memo... I just don't expect a DB like MongoDB to guarantee consistency. The whole story around NoSQL and the likes was to enable crazy horizontal scaling needed for the web. Phrases like "eventual consistency" flew around. It seems so logical - you lose consistency, gain scalability.

But somehow, people simply started using them everywhere? Assuming that these DBs are just like any other? And now, we're all bashing on MongoDB because it is - not consistent? What happened here? :)

NB that I do not wish to attack the OP - if MongoDB now claims to be consistent in any way, that deserves scrutiny. And these analyses are always a really interesting read. But the general tone in the developer community about MongoDB seems a bit irrational.

Re: Call Me Maybe: MongoDB Stale Reads

#118
post #104

Earlier quoted context omitted.

> Jepsen cannot prove that your software is safe, only prove that it isn't. This doesn't just apply to Jepsen but to all software tests. You only ever test a finite set of scenarios, so you can't really ever guarantee your software is 'safe'/bug-free - only that it does not fail in common/expected scenarios.

Depending on how large "software tests" is in your mind this can easily be not the case. Kyle often mentions the tool TLA+ which enables complete and total formal analysis of certain systems. This can be a test which is complete and therefore a positive proof of correctness. It's also not difficult to test smaller components of your system if you note that the state space here is small enough to be exhausted. This is…

Type systems and proofs have their limitations, if just because for a proof to be worth anything, I need a perfect description of what I want, and that doesn't make any sense outside of trivial cases.

I once worked with a relatively well known, prove everything developer that you might know by name. He's written books and everything. He built an algorithm to make a distributed system of equal peers figure out when it needed to start more nodes, or could shut some of them down. He wrote a proof, in code. He wrote a paper. When in production, the system would not work as advertised, and he blamed it on other pieces, because the algorithm was proven correct! So the problem stayed there for months.

After he left the company, I decided to figure the problem out, so I read through the proofs, the paper, and the code: All the single letter variables you could possibly want. I figured out that yes, the algorithm was flawless, as long as every operation in the system was atomic and instantaneous. Instead of the proof, I built a small simulator that didn't have such flawed assumptions, and got the exact same behavior as the production system. So the proof was perfect, as long as we made assumptions that are impossible in our universe. And the entire algorithm was less than 200 lines of code.

So whenever we have a reality that is difficult to model (and let me tell you, distributed systems fit the bill), dependent types will not save you, haskell or no haskell. Proofs will always be limited by your assumptions.

So while the tools you mention are nice. They hit the same limits as everything else we build. Whether to write a proof in idris, use generative testing, or just some example testing, is really all a tradeoff, but you will never escape from bad specification, as all specifications are bad.

Re: Call Me Maybe: MongoDB Stale Reads

#119

Earlier quoted context omitted.

Did you read the article? It's not about PostgreSQL in a distributed setup: > Even though the Postgres server is always consistent, the distributed system composed of the server and client together may not be consistent. It’s possible for the client and server to disagree about whether or not a transaction took place.

To me that's not surprising. If the client connection drops, you might not know whether the transaction committed or not... the only way to know is to reconnect and inspect to see what happened. If you want to avoid that kind of problem, use 2PC. Do you (or the author) see this as a bug, or just something that might surprise people who haven't thought through the guarantees?

It looks like you didn't read the article either -- it describes how Postgres is using 2PC, but 2PC doesn't stop the server from committed something that the client is unaware of through missing the final ACK.

Quoting the article:

> The 2PC protocol says that we must wait for the acknowledgement message to arrive in order to decide the outcome. If it doesn’t arrive, 2PC deadlocks. It’s not a partition-tolerant protocol. Waiting forever isn’t realistic for real systems, so at some point the client will time out and declare an error occurred. The commit protocol is now in an indeterminate state.

Re: Call Me Maybe: MongoDB Stale Reads

#120
post #117

I must admit, I always feel like I am missing something in these discussions. Like I didn't get some memo... I just don't expect a DB like MongoDB to guarantee consistency. The whole story around NoSQL and the likes was to enable crazy horizontal scaling needed for the web. Phrases like "eventual consistency" flew around. It seems so logical - you lose consistency, gain scalability. But somehow, people simply started…

"Eventual consistency" has a very particular meaning (when it's not being used as a buzzword). "Read uncommitted" doesn't even come close to the sorts of guarantees that people expect from an AP database. More importantly, MongoDB doesn't advertise itself as an AP database, it advertises itself as something you can use as the primary datastore for important information. Kyle has analyzed AP databases like Cassandra and Riak as well, and evaluates them according to their claims.
Post reply on HN