I can't help but think that this is an experience report covering "microservices-done-wrong considered harmful". There are definitely pitfalls to the microservice approach, but there are canonical answers to all of these issues. That said, the right approach is the one that fits with your team's disposition, skill-set, and experience level, so ditching microservices might well have been the correct choice in this case.
> Eventually, all of them were using different versions of these shared libraries. We could’ve built tools to automate rolling out changes, but at this point, not only was developer productivity suffering but we began to encounter other issues with the microservice architecture.
In the extreme case where you have 150 microservices and 3 devs, I think that spending a few days to build a tool that auto-updates your common deps and re-runs your tests would be a good investment. Or you could pay someone else to do this with a service like https://www.dependencies.io/. Handling common code is one of the known pain points in microservices, so it's worth tackling head-on. (Last I saw Netflix handles this by the rule "no services can share code unless it's by an open source library", which encourages common code to be thoughtfully packaged and released.)
> The additional problem is that each service had a distinct load pattern. Some services would handle a handful of events per day while others handled thousands of events per second. For destinations that handled a small number of events, an operator would have to manually scale the service up to meet demand whenever there was an unexpected spike in load.
I can see this being tricky to tune, and am hesitant to opine without knowing the details, but if you can fix the problem by bundling all the services into a single monolith (i.e. aggregating all load into N nodes), then you should also be able to fix the problem by using a cluster scheduler like k8s with equivalently sized nodes. As long as you don't have bursts that are an integer factor of your baseline system load, both approaches should work equivalently (to the first order). It sounds like they were running individual instance(s) per microservice, which isn't a good fit for very bursty services.
And as a bonus, with a cluster scheduler you get a number of primitives to do resource reservation, which you don't get for free if you're merging all of your services back inside a single monolith. This means the problem of back-pressure from a single misbehaving endpoint -- which was one of the reasons they moved to microservices in the first place -- will probably come up in some form down the road.
> Recall that the original motivation for separating each destination codebase into its own repo was to isolate test failures. However, it turned out this was a false advantage. Tests that made HTTP requests were still failing with some frequency. With destinations separated into their own repos, there was little motivation to clean up failing tests. This poor hygiene led to a constant source of frustrating technical debt.
I don't have anything to say here except... don't do this? If your tests are failing you should be fixing your tests (or removing them if they aren't adding value), not adding new integrations. Consistently-failing tests are a big warning sign that your CI/CD process is not in good shape, and a healthy CI/CD process is a strict precursor to doing microservices successfully.
> The outbound HTTP requests to destination endpoints during the test run was the primary cause of failing tests. Unrelated issues like expired credentials shouldn’t fail tests.
Your UTs shouldn't be hitting your external dependencies; the correct solution here is the one that they eventually landed on, i.e. to either record/replay real HTTP requests, or to mock out the HTTP responses manually. You still need real integration and smoke tests in a production-like environment to make sure that you've not missed a change in the remote API schema. IME this is one of the biggest challenges of working with external APIs, and I don't envy the task of maintaining 150 integrations, however this issue seems unrelated to microservices.