Just wanted to say thanks for this article. I have been exploring Debezium for capturing Pg changes at work. I don’t know a whole lot about Pg replication and it’s nice to hear the potential gotchas before moving to anything production-like.
Postgres, Kafka, and a mysterious 100 GB
21–24 of 24 posts
Re: Postgres, Kafka, and a mysterious 100 GB
#22The write activity every 5 minutes is standard Postgres checkpointing, the default value for checkpoint_timeout is 5 minutes. This is not limited to RDS. Background processes like vacuum and analyze also write to WAL.
Thanks, I didn't know that. Why do you think the Debezium docs call out this behavior specifically for RDS (https://debezium.io/documentation/reference/stable/connector...)? I'm wondering if there is any difference for RDS vs other hosting methods.
Re: Postgres, Kafka, and a mysterious 100 GB
#23Earlier quoted context omitted.
when turning on logical replication postgres keeps all wal files until they are also applied to the downstream consumer. if you don't use logical replication they get cleaned up by setting wal_keep_segments and max_wal_size (which have a sane value so there is nothing to bother) btw. a good idea is to keep these wal files or at least archive them with a basebackup, because it enables PITR recovery ( https://www.postg…
So this was 21 days of idle chatter? 1 WAL per 5 minute event, 16MB per WAL, 100GB/16MB = 6250 * 5m = 31,250m/1440m/day = 21ish days. Is that what happened?
In the post there is actually a graph showing that Transaction Logs filled the database in just 6 days (11/24 through 11/30). This doesn't add up to ~100 GB if we assume that each event takes up 16MB of space, but maybe each event took up even more storage than that.
Re: Postgres, Kafka, and a mysterious 100 GB
#24Had this exact thing happen in production when we turned off an audit DB replication slot. We got lucky and caught it before our entire app went down. It’s one of the many foot-guns we have found with Postgres.
What other footguns have you found ?