Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

pg_rewind in PostgreSQL 9.5

hlinnaka.iki.fi

Post reply on HN
Print view View original

51–60 of 63 posts

Re: pg_rewind in PostgreSQL 9.5

#51

Post by lmm »

afarrell wrote: post #15
moe wrote: post #9

This looks like a great tool, but it's also a sour reminder that replication still feels a lot like open heart surgery on postgresql. Why can't we just type "enslave 10.0.0.2" into psql and have the computer do the hard work? The machinery is "almost there" for a half a decade now. Who do we have to bribe (wink wink, nudge) to bring the UX into a state where crutches like pg_rewind are not needed?

> enslave "10.0.0.2" I know it is functionally immaterial, but boy howdy do I ever wish we'd chosen a better convention for how to refer to the relationship between these system components.

I find it's a useful heuristic for knowing whether someone can think clearly and abstractly. Kind of like the "all green birds have two heads" test.

Re: pg_rewind in PostgreSQL 9.5

#52

Post by davidgerard »

waitwaitwhay wrote: post #46
rosser wrote: post #32

Earlier quoted context omitted.

No, and nor did I mention "fencing". I said "minimum viable". No other open source RDBMS manages those things, so calling postgres out for lacking them is a little disingenuous. Replication isn't "inherently" anything. Making it a runtime thing is one way of doing it . Just because it's your preferred way (or — as often seems to be the case among people who share your opinion about how postgres does it — just how the…

> No other open source RDBMS manages those things, so calling postgres out for lacking them is a little disingenuous. Doesnt MySQL Cluster do this pretty well?

MySQL clustering is a goddamn PITA, just as MySQL replication is, regularly requiring you to take a plunger to unclog it. The quirks and bugs are myriad.

Re: pg_rewind in PostgreSQL 9.5

#53

Post by corford »

I don't get it. Why is it a problem taking a pg_basebackup of the new master to re-seed the old master (which is now a slave) and then promoting it to master again?

Or does pg_rewind offer a way to do this that doesn't require taking the current master offline when you promote a slave?

Re: pg_rewind in PostgreSQL 9.5

#54

Post by diroussel »

robd003 wrote: post #28
kenko wrote: post #26

Earlier quoted context omitted.

"man" is at least short for "manual". There's nothing essential about master/slave so why not just switch to primary/replica?

Because the master forces the slave to do it's bidding.

Not if the replica is pulling changes from the primary.

Re: pg_rewind in PostgreSQL 9.5

#55

Post by lobster_johnson »

moe wrote: post #47
lobster_johnson wrote: post #41

Earlier quoted context omitted.

> Rethinkdb, Redis, Riak, ElasticSearch all do. None of those are RDBMSes. Since they all lack schemas, constraints, strict consistency and ACID transactions, their replication logic can be much simpler. There are some very good reasons that true multimaster replication is nearly unknown among relational databases.

Since they all lack schemas, constraints, strict consistency and ACID transactions, their replication logic can be much simpler. Why would an RDBMS txlog be somehow harder to ship than any other db log? And how is that an excuse for crappy usability in any case?

Conflict management and consistency in general.

The products you mention have no interdependencies within relations or between relations, such as an RDBMS has, which means that they can just push streams of individual K/V pairs without having to maintain consistency between them.

Most of those projects only offer temporal consistency (last write wins) and the CRDTs offered (eg., Riak) are exceedingly simple compared to the complexities of managing uniqueness constraints or foreign key constraints across transactions.

Hinted handoff -- a common technique that mitigates split-brain data loss and increases availability in a shared scenario -- simply wouldn't work in a relational scenario because nobody would have any idea if the changes would apply when the target shard came back.

Projects like Postgres-XC (ActorDB also looks promising) have shown that multimaster replication is clearly possible in a relational database, but also that it's clearly a difficult problem to crack, because as has been discovered the last decade or so, consistency is the enemy of availability and partition tolerance.

Witness the very slow progress made by XC to retrofit multimaster replication into Postgres, and the many challenges (such as sequences) they have had to solve. Or the fact that it's pretty much alone in the open-source field in trying to solve this issue with a modern relational database. Even very mature, advanced, commercial solutions like Oracle RAC and Oracle Advanced Replication require a lot of configuration, careful design and monitoring to get right.

Note that I'm explicitly using the projects you mentioned as the benchmark here, ie. multimaster-replicated stores with automatic failover based on consensus protocols. You can build a simpler system — single master, consensus-based election, strict consistency guarantee across shards (reducing either availability or partition tolerance) — but it's still harder than you seem to think it is.

Re: pg_rewind in PostgreSQL 9.5

#56

Post by moe »

lobster_johnson wrote: post #55
moe wrote: post #47

Earlier quoted context omitted.

Since they all lack schemas, constraints, strict consistency and ACID transactions, their replication logic can be much simpler. Why would an RDBMS txlog be somehow harder to ship than any other db log? And how is that an excuse for crappy usability in any case?

Conflict management and consistency in general. The products you mention have no interdependencies within relations or between relations, such as an RDBMS has, which means that they can just push streams of individual K/V pairs without having to maintain consistency between them. Most of those projects only offer temporal consistency (last write wins) and the CRDTs offered (eg., Riak) are exceedingly simple compared…

Hinted handoff multimaster

What does multi-master have to do with anything?

The subject was making the postgres master/slave replication as easy to use as the aforementioned db's.

Yes, some of them use more advanced clustering models (multi-master) than postgres and are still easier to use. That doesn't exactly make postgres look better in the comparison.

Re: pg_rewind in PostgreSQL 9.5

#57

Post by lobster_johnson »

moe wrote: post #56
lobster_johnson wrote: post #55

Earlier quoted context omitted.

Conflict management and consistency in general. The products you mention have no interdependencies within relations or between relations, such as an RDBMS has, which means that they can just push streams of individual K/V pairs without having to maintain consistency between them. Most of those projects only offer temporal consistency (last write wins) and the CRDTs offered (eg., Riak) are exceedingly simple compared…

Hinted handoff multimaster What does multi-master have to do with anything? The subject was making the postgres master/slave replication as easy to use as the aforementioned db's. Yes, some of them use more advanced clustering models (multi-master) than postgres and are still easier to use. That doesn't exactly make postgres look better in the comparison.

Are you being deliberately obtuse? You explicitly mentioned four multimaster key/value stores:

  Rethinkdb, Redis, Riak, ElasticSearch all do.
  (them not being relational is irrelevant to replication)
I argued that those are not RDBMSes, and that their replication model was much simpler, something you still seem to be arguing against.

I don't disagree that Postgres's replication could be much nicer, but you're not articulating your position well. For one, bringing up these completely apples-to-orange comparisons makes no sense.

Re: pg_rewind in PostgreSQL 9.5

#58

Post by moe »

lobster_johnson wrote: post #57
moe wrote: post #56

Earlier quoted context omitted.

Hinted handoff multimaster What does multi-master have to do with anything? The subject was making the postgres master/slave replication as easy to use as the aforementioned db's. Yes, some of them use more advanced clustering models (multi-master) than postgres and are still easier to use. That doesn't exactly make postgres look better in the comparison.

Are you being deliberately obtuse? You explicitly mentioned four multimaster key/value stores: Rethinkdb, Redis, Riak, ElasticSearch all do. (them not being relational is irrelevant to replication) I argued that those are not RDBMSes, and that their replication model was much simpler, something you still seem to be arguing against. I don't disagree that Postgres's replication could be much nicer, but you're not artic…

Are you being deliberately obtuse?

You're the one being obtuse here.

I introduced four databases, three of which use more complex replication models than PG, all of which are easier to admin than PG. And I'm asking PG to make their (simpler) replication as easy to use as these other databases.

If that's too confusing then just use Redis for reference and ignore the other three. Redis is plain master/slave and spinning up a slave takes a single command ("SLAVEOF foo"). That's exactly how it should be in postgresql as well. Not more, not less.

Re: pg_rewind in PostgreSQL 9.5

#59

Post by lobster_johnson »

moe wrote: post #58
lobster_johnson wrote: post #57

Earlier quoted context omitted.

Are you being deliberately obtuse? You explicitly mentioned four multimaster key/value stores: Rethinkdb, Redis, Riak, ElasticSearch all do. (them not being relational is irrelevant to replication) I argued that those are not RDBMSes, and that their replication model was much simpler, something you still seem to be arguing against. I don't disagree that Postgres's replication could be much nicer, but you're not artic…

Are you being deliberately obtuse? You're the one being obtuse here. I introduced four databases, three of which use more complex replication models than PG, all of which are easier to admin than PG. And I'm asking PG to make their ( simpler ) replication as easy to use as these other databases. If that's too confusing then just use Redis for reference and ignore the other three. Redis is plain master/slave and spinn…

They can and do use different replication models because their data model is explicitly wildly different from Postgres. For example, Postgres couldn't support eventual consistency even if it wanted to. You're barking up the wrong tree.

As for Redis, sure, it has a simple master/slave system and Postgres could offer a similarly simple command UI that would let you issue a command instead of doing the current, trivial amount of setup. I don't disagree, but I think it's an inconsequential problem that affects a sysadmin/devops person about five minutes each year, and an issue that ultimately only deters dilettantes. The real problem to solve is dynamic promotion/demotion of masters and slave. It's not entirely trivial [1].

[1] http://redis.io/topics/sentinel

Re: pg_rewind in PostgreSQL 9.5

#60

Post by moe »

lobster_johnson wrote: post #59
moe wrote: post #58

Earlier quoted context omitted.

Are you being deliberately obtuse? You're the one being obtuse here. I introduced four databases, three of which use more complex replication models than PG, all of which are easier to admin than PG. And I'm asking PG to make their ( simpler ) replication as easy to use as these other databases. If that's too confusing then just use Redis for reference and ignore the other three. Redis is plain master/slave and spinn…

They can and do use different replication models because their data model is explicitly wildly different from Postgres. For example, Postgres couldn't support eventual consistency even if it wanted to. You're barking up the wrong tree. As for Redis, sure, it has a simple master/slave system and Postgres could offer a similarly simple command UI that would let you issue a command instead of doing the current, trivial…

Postgres couldn't support eventual consistency even if it wanted to. You're barking up the wrong tree.

And you're barking at a strawman. Nobody was talking about eventual consistency.

I don't disagree

You could do that less verbosely then.

The real problem to solve is dynamic promotion/demotion of masters

This whole thread is about how postgres hasn't even streamlined the manual promotion/demotion of one master in a simple master/slave pairing.

Post reply on HN