Designing for automation, however, involves significant service-model constraints. For example, some of the large services today depend upon database systems with asynchronous replication to a secondary, back-up server. Failing over to the secondary after the primary isn't able to service requests loses some customer data due to replicating asynchronously. However, not failing over to the secondary leads to service downtime for those users whose data is stored on the failed database server. Automating the decision to fail over is hard in this case since its dependent upon human judgment and accurately estimating the amount of data loss compared to the likely length of the down time. A system designed for automation pays the latency and throughput cost of synchronous replication. And, having done that, failover becomes a simple decision: if the primary is down, route requests to the secondary. This approach is much more amenable to automation and is considerably less error prone.
Bidirectional Replication is coming to PostgreSQL 9.6
21–30 of 71 posts
Re: Bidirectional Replication is coming to PostgreSQL 9.6
#22Running CitusDB with just 1 master made me nervous. They did talk about having multi-master replication as a belt and braces solution, but I don't know how far they got.
Thinking about this. Both being used may give you a 100% fully fault tolerant solution?
Re: Bidirectional Replication is coming to PostgreSQL 9.6
#23Re: Bidirectional Replication is coming to PostgreSQL 9.6
#24Looking forward to playing around with this. Native master-master replication is the only thing keeping me on MySQL.
Just curious, what Postgres features are you missing on MySQL? I had only used MySQL until a year or two ago, and wondered what I was missing since Postgres seems to get more love/hype from the developer community for whatever reason. Now using Postgres in production, there are few if any features that I notice our team using which don't exist in MySQL (maybe Json landed in Postgres first is one big one?). One thing…
hstore/jsonb/array/composite types https://www.postgresql.org/docs/8.1/static/rowtypes.html : array is often a good option to implement tag system
partial index: imagine you have a lot of "soft deleted" rows (i.e with a flag deleted turned to true), you can create an index that ignore them
index on expression: you often do request like "where date = today" , but you store timestamp precise to the second ? and you don't want to run date_trunc(your_column, 'day') , which it also a function not present in mysql..., everytime, nor you want to create a dedicated column for that only for the sake of performance, index on expression permit you to do that.
integrated full text search: you have a smallteam, and you don't feel like maintaining one more service for indexing and keeping in sync your search engine, here you are (of course it's not perfect but better than the option provided by MySQL)
table inheritance for partionning: you create one table "orders" , and you can easily partion them into "orders_2016" "orders_2015" etc. while still simply selecting things out of "orders"
constraints: your column "event_start" must be before "event_end", you can enforce that at the table level in PostgreSQL
text columns: in PostgreSQL don't worry with varchar(XXX) with XXX being the subject to flamewars (256 ? 500 ? 1000), the type "text" in postgresql is up to 2Go and as efficient as varchar()
uuid support: postgresql support uuid natively as primary keys (without needing to resort to a varchar ofcourse...)
And I've only talking about the advantage of PostgreSQL, not the strange defect of MySQL: for example that you can only have 1 column with a default timestamp, that your autoicrements will overflow silently taking back previous ids , a lot of things are only "warnings" (value not in an enum, fine I will insert null) that you will not see in your application code except if you really look hard for it.
Edit: I've used MySQL extensively and only started for now 2 years to use PostgreSQL, and though I have more knowledge in MySQL optimization and internals, and I don't consider it "bad", it's just 'so so', you will definitely be able to do whatever you want with it and it will not betray you hard, but PostgreSQL is just from an other league and will actively help you.
Re: Bidirectional Replication is coming to PostgreSQL 9.6
#25Does BDR have rules for primary key insertion conflicts? I have a (perhaps odd) situation where identical data is already being written to multiple servers. Currently handling with a custom replication mechanism.
Re: Bidirectional Replication is coming to PostgreSQL 9.6
#26Earlier quoted context omitted.
Just curious, what Postgres features are you missing on MySQL? I had only used MySQL until a year or two ago, and wondered what I was missing since Postgres seems to get more love/hype from the developer community for whatever reason. Now using Postgres in production, there are few if any features that I notice our team using which don't exist in MySQL (maybe Json landed in Postgres first is one big one?). One thing…
transactional DDL => if your application often has schema update and you use a tool like Doctrine for PHP / Alembic for python, when a downgrade or upgrade fail on MySQL in the middle became the create index was already taken by someone who "hot-fixed" the database and that now you're in an inconsistent state and you have to clean stuff by hand you will regret to not be on PostgreSQL where it will have simply rollbac…
Re: Bidirectional Replication is coming to PostgreSQL 9.6
#27I look forward to when this lands on PostgreSQL 9.7 without the need for an extension. But more so when I can also include the Citus DB extension. Running CitusDB with just 1 master made me nervous. They did talk about having multi-master replication as a belt and braces solution, but I don't know how far they got. Thinking about this. Both being used may give you a 100% fully fault tolerant solution?
https://www.postgresql.org/message-id/flat/CABUevEzT3RqJZR2i...
Re: Bidirectional Replication is coming to PostgreSQL 9.6
#28Please, please, please read the fine print and ensure you understand the design tradeoffs as well as your application's requirements before blindly using this.
The moment I heard multi-master I thought Paxos, Raft or maybe virtual synchrony. Hmm, nothing in the documentation. Maybe a new consensus protocol was written from scratch then? That should be interesting!
No, none of that either - this implementation completely disregards consistency and makes write conflicts the developer's problem.
From http://bdr-project.org/docs/stable/weak-coupled-multimaster....
* Applications using BDR are free to write to any node so long as they are careful to prevent or cope with conflicts
* There is no complex election of a new master if a node goes down or network problems arise. There is no wait for failover. Each node is always a master and always directly writeable.
* Applications can be partition-tolerant: the application can keep keep working even if it loses communication with some or all other nodes, then re-sync automatically when connectivity is restored. Loss of a critical VPN tunnel or WAN won't bring the entire store or satellite office to a halt.
Basically:
* Transactions are a lie
* Consistent reads are a lie
* Datasets will diverge during network partitioning
* Convergence is not guaranteed without a mechanism for resolving write conflicts
I am sure there are use-cases where the risk of this design is acceptable (or necessary), but ensure you have a plan for dealing with data inconsistencies!
Re: Bidirectional Replication is coming to PostgreSQL 9.6
#29Holy crap, I am scared! Please, please, please read the fine print and ensure you understand the design tradeoffs as well as your application's requirements before blindly using this. The moment I heard multi-master I thought Paxos, Raft or maybe virtual synchrony. Hmm, nothing in the documentation. Maybe a new consensus protocol was written from scratch then? That should be interesting! No, none of that either - thi…
I'd argue most non-financing applications would find these risks acceptable. This form of Multi-Master is what most people writing web-based applications actually are looking for. It simplifies having fail-over, at the costs you mentioned, but those aren't a major issues, especially if they're known upfront.
> * Datasets will diverge during network partitioning
> * Convergence is not guaranteed without a mechanism for resolving write conflicts
While this isn't ideal in a perfect world, it's workable for, again, web-based applications where consistency isn't usually required. Also, the rules are known http://bdr-project.org/docs/stable/conflicts-types.html
So yes, there are definitely workloads where this type of replication isn't appropriate, however, acting like there aren't any is blatantly ignoring many types of workloads.
Re: Bidirectional Replication is coming to PostgreSQL 9.6
#30Holy crap, I am scared! Please, please, please read the fine print and ensure you understand the design tradeoffs as well as your application's requirements before blindly using this. The moment I heard multi-master I thought Paxos, Raft or maybe virtual synchrony. Hmm, nothing in the documentation. Maybe a new consensus protocol was written from scratch then? That should be interesting! No, none of that either - thi…