This is a dupe of https://news.ycombinator.com/item?id=5912035.
Jepsen: Testing Partition Tolerance of PostgreSQL, Redis, MongoDB, Riak (2013)
11–16 of 16 posts
Re: Jepsen: Testing Partition Tolerance of PostgreSQL, Redis, MongoDB, Riak (2013)
#12For me choosing a database involves carefully considering the value of data that will be housed within. To what degree do I care about the data? How long would it take to recreate? What's the cost to my business if data is lost. Time and time again, I end up choosing PostgreSQL for ANY data that I care about (i.e. anything that is not simply 'in-flight data'.
From the point of view of a "devops" developer - who has done hard time in QA and DBA as well - this requirement changes drastically when performance and scaling enter the equation. The scale slides quickly from "lose no data, ever" to "we can lose a few seconds of transactions if it speeds up the web page" to "we can lose a lot of data and still be OK, as long as we're still online". Seconds, or even minutes worth o…
Wouldn't message queues help you with this? They can be easily scaled horizontally to handle any write load and correctly insert data to the database safe over time.
You can also shard your database to distribute write load.
Re: Jepsen: Testing Partition Tolerance of PostgreSQL, Redis, MongoDB, Riak (2013)
#13Earlier quoted context omitted.
From the point of view of a "devops" developer - who has done hard time in QA and DBA as well - this requirement changes drastically when performance and scaling enter the equation. The scale slides quickly from "lose no data, ever" to "we can lose a few seconds of transactions if it speeds up the web page" to "we can lose a lot of data and still be OK, as long as we're still online". Seconds, or even minutes worth o…
Wouldn't message queues help you with this? They can be easily scaled horizontally to handle any write load and correctly insert data to the database safe over time. You can also shard your database to distribute write load.
All of those absolutely help, at the cost of added complexity, additional points of failure, and hardware/VPS costs. And you still risk losing data, or at least data integrity.
Not to mention there are still hard limits on how quickly you can insert data into a database with 100% durability (which is, of course, impossible, but another topic entirely), and there are scales where even these mitigation tactics can't help you anymore (in particular, online casino games have this problem since they are persisting the state of multiple players very frequently).
Re: Jepsen: Testing Partition Tolerance of PostgreSQL, Redis, MongoDB, Riak (2013)
#14FoundationDB did an extensive test with Jepsen, with results published here: https://foundationdb.com/blog/call-me-maybe-foundationdb-vs-...
They are shipping a closed source black box and advertising their own demos and tests for a while. Just like this one. Not sure how much I would trust that more than just having a marketing statement saying "oh yeah we are the most stable, fastest, etc".
Re: Jepsen: Testing Partition Tolerance of PostgreSQL, Redis, MongoDB, Riak (2013)
#15For me choosing a database involves carefully considering the value of data that will be housed within. To what degree do I care about the data? How long would it take to recreate? What's the cost to my business if data is lost. Time and time again, I end up choosing PostgreSQL for ANY data that I care about (i.e. anything that is not simply 'in-flight data'.
From the point of view of a "devops" developer - who has done hard time in QA and DBA as well - this requirement changes drastically when performance and scaling enter the equation. The scale slides quickly from "lose no data, ever" to "we can lose a few seconds of transactions if it speeds up the web page" to "we can lose a lot of data and still be OK, as long as we're still online". Seconds, or even minutes worth o…
Out of interest, What hardware have you tried running your Postgres clusters on?
Re: Jepsen: Testing Partition Tolerance of PostgreSQL, Redis, MongoDB, Riak (2013)
#16FoundationDB did an extensive test with Jepsen, with results published here: https://foundationdb.com/blog/call-me-maybe-foundationdb-vs-...
They are shipping a closed source black box and advertising their own demos and tests for a while. Just like this one. Not sure how much I would trust that more than just having a marketing statement saying "oh yeah we are the most stable, fastest, etc".
That and the quite low limits for transaction duration (5 seconds!) make FoundationDB a no-go for me.
Closed source + very narrow limits due to system design makes this a very hard sell.
It's a pitty because they do seem to have some decent ideas in there. I like the layers that build more complex data models upon a transactional distributed key-value store.