My thoughts about Fly.io (so far) and other newish technology I'm getting into

There's a very solid solution to this that isn't as widely known as it should be.

Read after write consistency is extremely important. If a user makes an edit to their content and then can't see that edit in the next page they load they will assume things are broken, and that the site has lost their content. This is really bad!

The best fix for this is to make sure that all reads from that user are directed to the lead database for a short period of time after they make an edit.

The Fly replay header is perfect for this. Here's what to do:

Any time a user performs a write (which should involve a POST request), set a cookie with a very short time expiry - 5s perhaps, though monitor your worst case replica lag to pick the right value.

I have trust issues with clocks in user's browsers, so I like to do this by including a value of the cookie that's the server-time when it should expire.

In your application's top-level middleware, look for that cookie. If a user has it and the court time has not been reached yet, send a Fly replay header that internally redirects the request to the lead region.

This guarantees that users who have just performed a write won't see stale data from a lagging replica. And the implementation is a dozen or so lines of code.

Obviously this won't work for every product - if you're building a chat app where every active user writes to the database every few seconds implementing this will send almost every piece of traffic to your leaders leaving your replicas with not much to do.

But if your application fits the common pattern where 95% of traffic are reads and only a small portion of your users are causing writes at any one time I would expect this to be extremely effective.

Fly replay headers are explained in detail here: https://fly.io/blog/globally-distributed-postgres/

> It seems like this would add a whole new class of bugs, like “I just submitted a form to change a setting and when the page reloaded, it still showed my previous value in the form” – since the write hadn’t propagated to the local read replica yet.

Elixir is distributed out of the box, so nodes can message each other. This allowed us to easily ship a `fly_postgres_elixir` library that guarantees read-your-own-writes: https://github.com/superfly/fly_postgres_elixir

It does this by sending writes to the primary region over RPC (via distributed elixir). The write is performed on a primary instance adjacent to the DB, then the result, and the postgres log-sequence-number, is sent back to the remote node. When the library gets a result of the RPC write, it blocks locally until its local read replica matches an LSN >= write LSN, then the result is returned to the caller

This gives us read-your-own-writes for the end-user, and the calling code remains unchanged for standard code paths. This doesn't solve all classes of race conditions – for example you may broadcast a message over Phoenix.PubSub that causes a read on the remote node for data that isn't yet replicated, but typically you'd avoid an N query problem from pubsub in general by populating the data in the message on the publisher beforehand.

There's no completely avoiding the fact you have a distributed system where the speed of light matters, but it's Fly's (and Phoenix's) goal to push those concerns back as far as possible. For read heavy apps, or apps that use caching layers for reads, developers already face these kinds of problems. If you think of your read-replicas as cache with a convenient SQL interface, you can avoid most foot guns.

I'm happy to answer other questions as it relates to Phoenix, Fly or what Phoenix + Fly enables from my perspective.

I've looked at implementing this in the past and always found it to be SO difficult that the benefit would not be worth the cost.

Fly has changed that equation for me. It has moved this problem from "I'd love to do it if I could but it's just too hard" to "This is a thing I could do with small enough engineering effort that it would be worthwhile".

This is my favourite type of technology: I love things that move something from the "too expensive" to the "now feasible to implement" bucket!