Don't add a read replica until you've read this
1–7 of 7 posts
Re: Don't add a read replica until you've read this
#2I work with Johanna (author) on the incident team. It’s been pretty cool watching this roll out, been particularly great that I’ve barely had to think about the replicas when working with the code.
The wiring to route to either replica or primary is very transparent. Seen this done several times before and it’s never been quite as easy to have your code just do the Right Thing.
Re: Don't add a read replica until you've read this
#3I work with Johanna (author) on the incident team. It’s been pretty cool watching this roll out, been particularly great that I’ve barely had to think about the replicas when working with the code. The wiring to route to either replica or primary is very transparent. Seen this done several times before and it’s never been quite as easy to have your code just do the Right Thing.
Also work with Johanna and agree this was fun to work on!
Biggest win for us now IMO is that we can just horizontally scale reads. Instead of trying to play whackamole with micro-optimisiations, we just buy more computers. (Optimisation still has its place obvs!) And, the way this is set up, load balancing across multiple replicas can also be completely hidden from callers.
Neat
Re: Don't add a read replica until you've read this
#4Its a shame they didn't go with a Bladerunner replicant pun for the title. "I've seen database performance you wouldn't believe"
Re: Don't add a read replica until you've read this
#5Disclaimer: I worked with Johanna on this
I had a lot of fun “long tail whackamoling” our way through edge queries that our model of “this is eligible for a read replica” wasn’t accurately catching. Honestly just one of those “keep reading traces and working through them” development processes that just keeps drive a graph down and to the right.
Re: Don't add a read replica until you've read this
#6Disclosure: I work at incident.io with the author. My favourite part of how we implemented this is that consistency stopped being something engineers had to continuously think about and became part of the Go context object in our codebase; so any subsequent write in the same context taints without them having to think, and tainted reads go to the primary.
Re: Don't add a read replica until you've read this
#7I work with Johanna (author) on the incident team. It’s been pretty cool watching this roll out, been particularly great that I’ve barely had to think about the replicas when working with the code. The wiring to route to either replica or primary is very transparent. Seen this done several times before and it’s never been quite as easy to have your code just do the Right Thing.
Also work with Johanna and agree this was fun to work on! Biggest win for us now IMO is that we can just horizontally scale reads. Instead of trying to play whackamole with micro-optimisiations, we just buy more computers. (Optimisation still has its place obvs!) And, the way this is set up, load balancing across multiple replicas can also be completely hidden from callers. Neat
You guys sound like cool guys.