Microservices Killed Our Startup. Monoliths Would've Saved Us

But that starts to fall down too any time too many people are talking about software they aren’t responsible for deploying or fixing.

Microservices solve a logistical problem. Rob wants to push code every two days. Steve wants to push every three. Thom deals with business who wants to release at whim and preferably within a few hours. Their commissions and bonuses are not reduced by how much chaos they case the engineering team. It’s an open feedback loop.

As you add more employees they start tripping over each other on the differences between trunk and deployed. Thats when splitting into multiple services starts to look attractive. Unfortunately they create their own weather and so if you can use process to delay this point you’re gonna be better off.

Everyone eventually merges code they aren’t 100% sure about. Some people do it all the time. However microservices magnify this because it’s difficult to test changes that cross service boundaries. You think you have it right but unless you can fit the entire system onto one machine, you can’t know. And distributed systems usually don’t concern themselves with whether the whole thing will fit onto a dev laptop.

So then you have code in preprod you are pretty sure will work but aren’t completely sure. Stack enough “pretty sure”s over time and as team sizes grow and you’re gonna have incidents on the regular. Separate deployment reduces the blast radius, but doesn’t eliminate it. Feature toggles reduce it more than an order of magnitude, but that still takes you from problems every week to a couple a year. Which in high SLA environments still makes people cranky.

You know a team has lost the architectural plot when their answer for all performance problems is more caching. And once you add caching it’s hard to sell any other sort of improvements because the caching poisons the perf analysis.

Their solution took forever because the system was less deterministic than we even knew. They were starting to wrap it up when I went on a tear cleaning up low level code that was nickel and diming us. By the time they launched they were looking at achieving half of the response time improvement they were looking for, in twice the time they estimated to do so. And they cheated. They making two requests about 10% of the time, which made the p50 time into a lie, because two smaller requests pull down the average but not the cost per page load. But I scooped them and made the slow path faster, undercutting another 25% of their perf improvements.

I ended up doing more to improve the Little’s Law situation in three months of working on it half time than they did in two man years. And still nothing changed. They are now owned by a competitor. That I believe shut down almost all of their services.