Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

pg_rewind in PostgreSQL 9.5

hlinnaka.iki.fi

Post reply on HN
Print view View original

21–30 of 63 posts

Re: pg_rewind in PostgreSQL 9.5

#21

Post by neilk »

afarrell wrote: post #15
moe wrote: post #9

This looks like a great tool, but it's also a sour reminder that replication still feels a lot like open heart surgery on postgresql. Why can't we just type "enslave 10.0.0.2" into psql and have the computer do the hard work? The machinery is "almost there" for a half a decade now. Who do we have to bribe (wink wink, nudge) to bring the UX into a state where crutches like pg_rewind are not needed?

> enslave "10.0.0.2" I know it is functionally immaterial, but boy howdy do I ever wish we'd chosen a better convention for how to refer to the relationship between these system components.

My proposal: muse and scribe. It's evocative and if you know the old terminology the initials are the same.

Re: pg_rewind in PostgreSQL 9.5

#22

Post by olefoo »

afarrell wrote: post #15
moe wrote: post #9

This looks like a great tool, but it's also a sour reminder that replication still feels a lot like open heart surgery on postgresql. Why can't we just type "enslave 10.0.0.2" into psql and have the computer do the hard work? The machinery is "almost there" for a half a decade now. Who do we have to bribe (wink wink, nudge) to bring the UX into a state where crutches like pg_rewind are not needed?

> enslave "10.0.0.2" I know it is functionally immaterial, but boy howdy do I ever wish we'd chosen a better convention for how to refer to the relationship between these system components.

I try to use 'primary' and 'replica'; but depending on who I'm talking to I find myself slipping back to the Master/Slave terminology.

And to the OP, replication is a complex process fraught with security risks. Take about 10 minutes thinking about just roles and authentication in the context of your proposed syntax and what you would need to set up ahead of time to make it work.

The current method of starting with a pg_basebackup on the replica is nearly as simple. Even if it does require that you do some minor configuration on the master. It's not nearly open heart surgery.

Re: pg_rewind in PostgreSQL 9.5

#23

Post by moe »

olefoo wrote: post #22
afarrell wrote: post #15

Earlier quoted context omitted.

> enslave "10.0.0.2" I know it is functionally immaterial, but boy howdy do I ever wish we'd chosen a better convention for how to refer to the relationship between these system components.

I try to use 'primary' and 'replica'; but depending on who I'm talking to I find myself slipping back to the Master/Slave terminology. And to the OP, replication is a complex process fraught with security risks. Take about 10 minutes thinking about just roles and authentication in the context of your proposed syntax and what you would need to set up ahead of time to make it work. The current method of starting with a…

what you would need to set up ahead of time to make it work

Nothing except a shared secret or keypair. Other databases (rethinkdb, redis, riak etc.) show how it's done.

Re: pg_rewind in PostgreSQL 9.5

#24

Post by snuxoll »

It's nice to see a lot of work being put into mirroring replication in the PostgreSQL 9.0 line, but until better admin tools are there I'll probably just keep using corosync and pacemaker with a shared fiber channel volume for clustering. Sure, it's cold standby, but it only takes a couple seconds for a standby node to come up and I just have to keep my WAL backups like normal for recovery.

Re: pg_rewind in PostgreSQL 9.5

#25

Post by amitlan »

rdtsc wrote: post #8

ActorDB is an interesting project that operates on distributed SQLite database. It tries to provide clustering between instances and it does it by continuously replicating the WAL between nodes. I am not affiliated with the project, but just saw it the other day and thought it was a pretty cool pattern: http://www.actordb.com/ Here is the excerpt from their description page: --- Actors are replicated using the Raft d…

I hope something like BDR project evolves in a UX-centric (too!) direction.

[0] http://blog.2ndquadrant.com/dynamic-sql-level-configuration-...

[1] https://wiki.postgresql.org/wiki/BDR_User_Guide

Re: pg_rewind in PostgreSQL 9.5

#26

Post by kenko »

moe wrote: post #18
afarrell wrote: post #15

Earlier quoted context omitted.

> enslave "10.0.0.2" I know it is functionally immaterial, but boy howdy do I ever wish we'd chosen a better convention for how to refer to the relationship between these system components.

I know it is functionally immaterial There's nothing wrong with Master/slave. We also read man pages, shove male plugs into female ports, fork and kill children, and use dozens of other odd words to describe perfectly harmless things.

"man" is at least short for "manual".

There's nothing essential about master/slave so why not just switch to primary/replica?

Re: pg_rewind in PostgreSQL 9.5

#27

Post by rosser »

moe wrote: post #9

This looks like a great tool, but it's also a sour reminder that replication still feels a lot like open heart surgery on postgresql. Why can't we just type "enslave 10.0.0.2" into psql and have the computer do the hard work? The machinery is "almost there" for a half a decade now. Who do we have to bribe (wink wink, nudge) to bring the UX into a state where crutches like pg_rewind are not needed?

A thing people consistently seem not to get when bitching about PostgreSQL (streaming) replication is that, by design, it's slave pulls instead of master pushes. There are Very Good Reasons for that.

And it's really not that hard. MVPRC (Minimum Viable PostgreSQL Replication Configuration):

  Master:
    postgresql.conf:
      wal_level = 'hot_standby'
      max_wal_senders = N
    pg_hba.conf:
      host replication all A.B.C.D/E md5
  Slave:
    postgresql.conf:
      hot_standby = 'on'
    recovery.conf
      standby_mode = 'on'
      primary_conninfo = 'host=W port=X user=Y password=Z'
Literally six lines of "directive = value". (Well, five plus the HBA rule.)

Then you use pg_basebackup, and start the slave.

EDIT: Decided to be specific instead of handwavy.

Re: pg_rewind in PostgreSQL 9.5

#28

Post by robd003 »

kenko wrote: post #26
moe wrote: post #18

Earlier quoted context omitted.

I know it is functionally immaterial There's nothing wrong with Master/slave. We also read man pages, shove male plugs into female ports, fork and kill children, and use dozens of other odd words to describe perfectly harmless things.

"man" is at least short for "manual". There's nothing essential about master/slave so why not just switch to primary/replica?

Because the master forces the slave to do it's bidding.

Re: pg_rewind in PostgreSQL 9.5

#29

Post by moe »

kenko wrote: post #26
moe wrote: post #18

Earlier quoted context omitted.

I know it is functionally immaterial There's nothing wrong with Master/slave. We also read man pages, shove male plugs into female ports, fork and kill children, and use dozens of other odd words to describe perfectly harmless things.

"man" is at least short for "manual". There's nothing essential about master/slave so why not just switch to primary/replica?

Because there is no reason to screw with common terminology to begin with.

Neither Master nor Slave is a derogatory word. And using the concept of slavery to describe a relationship between inanimate objects in a technical context is not an endorsement of such practices between humans.

Re: pg_rewind in PostgreSQL 9.5

#30

Post by jeffdavis »

moe wrote: post #9

This looks like a great tool, but it's also a sour reminder that replication still feels a lot like open heart surgery on postgresql. Why can't we just type "enslave 10.0.0.2" into psql and have the computer do the hard work? The machinery is "almost there" for a half a decade now. Who do we have to bribe (wink wink, nudge) to bring the UX into a state where crutches like pg_rewind are not needed?

I'd summarize the replication story of postgres to be: "harder to use but easier to understand".

PostgreSQL generally builds things from the bottom up -- first working really hard to get the fundamental infrastructure right, and then the utility starts falling into place.

This has a few advantages:

* The fundamental infrastructure tends to be more general, so many different functional improvements come out quickly after the infrastructure is in place.

* It forces everyone involved to think through the edge cases first, leading to a more robust feature and a better-tested feature.

* When things go wrong, all of the parts make more sense individually, so it's easier to understand what happened and fix it.

* Eventually, the usability issues are addressed in a way that's not dependent on a lot of black magic.

Anecdotally, when things go wrong in $MAGICALLY_REPLICATING_DB, they tend to go very wrong, very fast. I am skeptical by nature, so I generally assume that $MAGICALLY_REPLICATING_DB glosses over a lot of the finer points, which don't get exercised in a normal testing environment. I haven't presented any facts here, so you may disagree.

A lot of people share my skepticism, and are willing to put up with a few rough edges in the usability as long as the pieces make sense and they feel they can understand and trust them. I was at a presentation by Instagram, and replication was one of the primary reasons they went with postgres. They used a tiny engineering team (one and a half engineers working on the database) to scale to tens of millions of active users (also using sharding; another thing that is harder to use but easier to understand in postgres).

Post reply on HN