Read-Only Mode for Better Rails Downtime • Hacker News Board

Re: Read-Only Mode for Better Rails Downtime

#11

Post by xfalcox » Wed, Oct 14, 2020, 7:30 PM UTC

christoomey wrote: post #10

xfalcox wrote: post #8

Earlier quoted context omitted.

PostgreSQL upgrades are indeed one of the use cases we have with the gem, the way we do it is: 1. Have app configured to connect to both main and replica. 2. Connect to the rails console and tell the app to stay in read only mode until told otherwise. 3. Disable replication 4. Upgrade main to new PostgreSQL version 5. Tell the app to move back to read-write mode 6. Re-create the replica This flow helped us do hundred…

Oh that's interesting, thanks for the additional detail! I'm intrigued by what you mean when you say: "tell the app to stay in read only mode until told otherwise". Does that mean use the read-only replica connection during that time? If so, did you have to configure some error handling for that timeframe?

> Does that mean use the read-only replica connection during that time?

Yes!

https://github.com/discourse/discourse/blob/721ee3642558d960...

> If so, did you have to configure some error handling for that timeframe?

Since this was an early goal in the project, the controllers are mostly away of the read-only mode already and know how to deal with it is most places.

Re: Read-Only Mode for Better Rails Downtime

#12

Post by christoomey » Wed, Oct 14, 2020, 9:06 PM UTC

xfalcox wrote: post #11

christoomey wrote: post #10

Earlier quoted context omitted.

Oh that's interesting, thanks for the additional detail! I'm intrigued by what you mean when you say: "tell the app to stay in read only mode until told otherwise". Does that mean use the read-only replica connection during that time? If so, did you have to configure some error handling for that timeframe?

> Does that mean use the read-only replica connection during that time? Yes! https://github.com/discourse/discourse/blob/721ee3642558d960... > If so, did you have to configure some error handling for that timeframe? Since this was an early goal in the project, the controllers are mostly away of the read-only mode already and know how to deal with it is most places.

I've added a note at the bottom of the post as your gem seems like a great option for this sort of functionality. Thanks again for sharing!

Re: Read-Only Mode for Better Rails Downtime

#13

Post by stubish » Thu, Oct 15, 2020, 1:05 PM UTC

A similar approach, for both horizontally scaling your db and automatic read-only mode. Ensure you are following the rule to only attempt writes in POST requests. POST requests get their queries sent to the primary db. GET requests send db queries to a read-only standby, unless the client sent a POST request recently (storing the last POST timestamp in a cookie or a session db, and the value for 'recently' is the current database replication lag). Catch any connection failures for requests using the primary db, and retry on a read-only standby. Catch db write failure to a read-only standby and replace with a 'down for maintenance' page.

We retrofitted the above behavior to an ~400k line monolithic Python app with minimal changes a bit over 10 years ago (the web framework was thankfully agreeable), shed over half the load on the master day 1, and could happily bounce the master db or do schema updates at will without needing to touch the appservers. Plug it into your CI system and you can do both automatic deployments and database updates, often with unnoticeable downtime if you are clever about your database patches.