Logical replication and decoding for Cloud SQL for PostgreSQL

Post by gunnarmorling » Sat, Jun 05, 2021, 10:04 PM UTC

mrbungie wrote: post #5

pyrophane wrote: post #2

Hijacking this thread a bit for a related question. Anyone have a solution for replicating data from Cloud SQL Postgres to BigQuery that they like? Have been shopping around for a good way to do this, ideally with the ability to capture deletions and schema changes. Have looked at Fivetran but it seems expensive for this use case, and won’t capture deletions until they can support logical replication.

An option, albeit a bit complex would be: Debezium => Kafka => Parquet in Google Cloud Storage => BigQuery external queries.

Post by cpursley » Sat, Jun 05, 2021, 10:33 PM UTC

btown wrote: post #14

This is awesome! For comparison, this is also supported on Amazon RDS, so AFAICT this opens up the possibility of near-zero-downtime streaming migrations between the two cloud providers: https://aws.amazon.com/blogs/database/using-logical-replicat... Also, it enables a really cool pattern of change data capture, which allows you to capture "normal" changes to your Postgres database as events that can be fed to e.g. K…

Post by Diggsey » Sat, Jun 05, 2021, 10:55 PM UTC

jacobkg wrote: post #22

Diggsey wrote: post #16

Earlier quoted context omitted.

Also, if you use Cloud SQL in HA mode, it may still go down randomly for up to 90s with no warning or entry in the operation log, and this is considered expected behaviour. Here is a direct quote from google support when we contacted them about our database going down outside of our scheduled maintenance window: > As I mentioned previously remember that the maintenance window is preferred but there are time-sensitive…

That’s troubling. In fairness, when I last used RDS (2018) we had 9 databases running and we averaged about one database failover per month, with about 2-3 minutes of downtime per incident. I never got a satisfactory answer from support other than that this was a thing that sometimes happened.

Post by jacobkg » Sat, Jun 05, 2021, 11:01 PM UTC

Diggsey wrote: post #25

jacobkg wrote: post #22

Earlier quoted context omitted.

That’s troubling. In fairness, when I last used RDS (2018) we had 9 databases running and we averaged about one database failover per month, with about 2-3 minutes of downtime per incident. I never got a satisfactory answer from support other than that this was a thing that sometimes happened.

To be clear: there is no failover happening, even though we do have a failover instance. If there was, we could at least detect that something had happened after the fact! The Cloud SQL failover only occurs in certain circumstances, and in all our time using Cloud SQL the failover has not once kicked in automatically (despite many outages). In fact, one of our earliest support issues was that the "manual failover" bu…

Post by koolba » Sat, Jun 05, 2021, 11:15 PM UTC

jacobkg wrote: post #9

Glad to see that they are adding features but I wish the pace was faster. We have been using Cloud SQL for Postgres and overall it is good but there are a number of glaring and frustrating feature holes. The two top of mind for me are: 1) No way to force SSL connections without enforcing two-way SSL (which is a huge pain and not supported by all the clients we use). This is literally just a Postgres config option but…