Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres, Kafka, and a mysterious 100 GB

mattritter.me

Post reply on HN
Print view View original

11–20 of 24 posts

Re: Postgres, Kafka, and a mysterious 100 GB

#11

Post by tbrock »

We ran into this too, i actually think its a terrible postgres default. Logical replication slots should have timeouts where if you haven't read the WAL record in say 24 hours it should be dropped. Make it configurable and set a sane default, problem solved.

You'd have to resync the followers/secondaries, big deal, its way better than the primary going down because its disk filled up. This failure mode is awful. On RDS its relatively painless because you can snap fingers and have more disk but if you are running it yourself? Good luck.

In practice, mongodb’s oplog mechanism, for example, which acts as a circular buffer with a set size is a much more tolerant implementation. If the oplog rolls over befrore you've read it just resync but at most its taken up 10% of your disk.

Re: Postgres, Kafka, and a mysterious 100 GB

#12

Post by Izkata »

tbrock wrote: post #11

We ran into this too, i actually think its a terrible postgres default. Logical replication slots should have timeouts where if you haven't read the WAL record in say 24 hours it should be dropped. Make it configurable and set a sane default, problem solved. You'd have to resync the followers/secondaries, big deal, its way better than the primary going down because its disk filled up. This failure mode is awful. On R…

> Logical replication slots should have timeouts where if you haven't read the WAL record in say 24 hours it should be dropped. Make it configurable and set a sane default, problem solved.

In other words, replication without using replication slots. Postgres has had it for a long time, although the limit is set in disk usage rather than time.

(unless specifically logical replication and/or debezium don't support this?)

Re: Postgres, Kafka, and a mysterious 100 GB

#13

Post by [deleted] »

tbrock wrote: post #11

We ran into this too, i actually think its a terrible postgres default. Logical replication slots should have timeouts where if you haven't read the WAL record in say 24 hours it should be dropped. Make it configurable and set a sane default, problem solved. You'd have to resync the followers/secondaries, big deal, its way better than the primary going down because its disk filled up. This failure mode is awful. On R…

[deleted]

Re: Postgres, Kafka, and a mysterious 100 GB

#14

Post by milanogc »

nja wrote: post #6

Debezium is a useful tool, but requires a lot of babysitting. If the DB connection blips or DNS changes (say, if you just rebuilt your prod db), or in some other cases, it'll die and present this exact problem. Fortunately, it's easy to enable a "heartbeat" topic to alert on to make sure it can be restarted before the db disk fills (of course, db size growth alerts are critical too). We've found that it's worth it fo…

How do you deal with late commits that have smaller identifiers. Using the incrementing mode those records will be skipped.

Re: Postgres, Kafka, and a mysterious 100 GB

#15

Post by voganmother42 »

Izkata wrote: post #12
tbrock wrote: post #11

We ran into this too, i actually think its a terrible postgres default. Logical replication slots should have timeouts where if you haven't read the WAL record in say 24 hours it should be dropped. Make it configurable and set a sane default, problem solved. You'd have to resync the followers/secondaries, big deal, its way better than the primary going down because its disk filled up. This failure mode is awful. On R…

> Logical replication slots should have timeouts where if you haven't read the WAL record in say 24 hours it should be dropped. Make it configurable and set a sane default, problem solved. In other words, replication without using replication slots. Postgres has had it for a long time, although the limit is set in disk usage rather than time. (unless specifically logical replication and/or debezium don't support this…

Postgres replication slots support a max size (since 13) specifically to deal with this scenario (prior to that needed to setup monitoring/intervention outside the db)

Re: Postgres, Kafka, and a mysterious 100 GB

#16

Post by nieve »

asjfkdlf wrote: post #2

Had this exact thing happen in production when we turned off an audit DB replication slot. We got lucky and caught it before our entire app went down. It’s one of the many foot-guns we have found with Postgres.

I'm not sure telling a replication system to keep copies of all changes without limits until they're replicated and then not letting them replicate is much of a footgun. You're getting exactly what you asked for and any system with the setup is going to eventually start filling up space if the replication or connector goes down. It's like complaining that your kitchen floor got wet when when you plugged the sink and left it running - it's unpleasant surprise, maybe you wanted a sink with an emergency overflow outlet, but it's the obvious outcome.

Re: Postgres, Kafka, and a mysterious 100 GB

#17

Post by j16sdiz »

asjfkdlf wrote: post #2

Had this exact thing happen in production when we turned off an audit DB replication slot. We got lucky and caught it before our entire app went down. It’s one of the many foot-guns we have found with Postgres.

Depends on your audit requirement, halting the operation when audit is unavailable maybe desirable.

Re: Postgres, Kafka, and a mysterious 100 GB

#18

Post by gregplaysguitar »

I heard an interesting comment recently from Derek Collison (creator of NATS[1]) that durability and delivery requirements can have the unwanted side-effect that one consumer can adversely impact all the others. It didn’t immediately make sense then, but this seems like a succinct illustration of the point!

[1] https://NATS.io

Re: Postgres, Kafka, and a mysterious 100 GB

#19

Post by gunnarmorling »

nja wrote: post #6

Debezium is a useful tool, but requires a lot of babysitting. If the DB connection blips or DNS changes (say, if you just rebuilt your prod db), or in some other cases, it'll die and present this exact problem. Fortunately, it's easy to enable a "heartbeat" topic to alert on to make sure it can be restarted before the db disk fills (of course, db size growth alerts are critical too). We've found that it's worth it fo…

Which version of Debezium was it you encountered these issues with? Connection losses should not be a problem with any current version, as the connectors will restart automatically in that case.

Agreed though that monitoring should be in place, so to be notified upon failed connectors early on (could be based on the heartbeat topic, but there's also JMX metrics which can be exposed to Prometheus/Grafana, and/or health checks could be set-up based on the connector's status as exposed via the Kafka Connect REST API).

On the matter of disk growth, there's no silver bullet here. Some people will want to make 100% sure that never ever events are missed, which implies the replication slot must hold onto those WAL segments while it's not read (this is not specific to Debezium btw.). Others may be willing to accept missing events if the slot isn't read long enough, so those WAL segments can be discarded. In recent Postgres versions, a max size (or age, not sure) can be configured for a replication slot, so it's a matter of configuration which behavior you want.

In any case, a connector downtime for longer than say a few hours is something that should show up as an alert, allowing to take action.

Disclaimer: I work on Debezium

Re: Postgres, Kafka, and a mysterious 100 GB

#20

Post by higeorge13 »

milanogc wrote: post #14
nja wrote: post #6

Debezium is a useful tool, but requires a lot of babysitting. If the DB connection blips or DNS changes (say, if you just rebuilt your prod db), or in some other cases, it'll die and present this exact problem. Fortunately, it's easy to enable a "heartbeat" topic to alert on to make sure it can be restarted before the db disk fills (of course, db size growth alerts are critical too). We've found that it's worth it fo…

How do you deal with late commits that have smaller identifiers. Using the incrementing mode those records will be skipped.

This and not being able to track deleted records requires a solution like debezium.
Post reply on HN