The Death of Microservice Madness in 2018

What would be an example of a situation where the backend dev needs to change a field in the database?

Once when starting a new gig I inherited a "microservices" architecture.

They were having performance problems and "needed" to migrate to microservices. They developed 12 seperate applications, all in the same repo, deployed independently it's own JVM. Of course if you were using microservices, you needed docker as well, so they had also developed a giant docker container containing all 12 microservices which they deployed to a single host (all managed by supervisord). Of course since they had 12 different JVM applications, the services needed a host with at least 9GiB of RAM so they used a larger instance. Everything was provisioned manually by the way because there was no service discovery or container orchestration - just a docker container running on a host (an upgrade from running the production processes in a tmux instance). What they really had was a giant monolithic application with a complicated deployment process and an insane JVM overhead.

Moving to the larger instance likely solved the performance issues. In place they now had multiple over provisioned instances (for "HA"), and combined with other questionable decisions, were paying ~100k/year for a web backend that did no more than ~50 requests/minute at peak. But hey at least they were doing real devops like Netflix.

For me, I've become a bit more aware of cargo cult development. I can't say I'm completely immune to cargo cult driven development either (I once rewrote an entire Angular application in React because "Angular is dead") so it really opened my eyes how I could also implement "solutions" without truly understanding why they are useful.

That's how we used to build systems back in the 3-tier architecture (DB, app server, UI) days. Usually backend servers would be hooked up to a workflow engine or message bus like Celery, JMQ, or plain old crontab to do batch tasks.

It's not a bad system - it worked for a technology generation, from around 1997-2012 - but it's somewhat limited in the problem domains it can solve effectively. Anything real-time or with high change volume becomes problematic, because each service needs to poll the DB for work to do. The DB can quickly become a scaling bottleneck. It's not as well-suited to thick clients, because you need an app server in front of the DB anyway to check & validate requests so the entire world doesn't get access to all your data.

On the plus side, state coherency becomes much easier, because all state is in the DB and you can use transactions to update it atomically. I also disagree with the posters who say that updates & versioning are a nightmare: they are, but updates are a nightmare in any distributed system, and sticking an app server in front of the DB just makes it easier to know when you've made a breaking change, it doesn't make the change itself easier. I've worked on large (= Google) protobuf-based SOAs before, and changing anything that touches backend protocols or storage is difficult regardless of which architecture you use.

I'm a big fan of the "solve the problem first, scale the architecture later" approach. For your first version, just store everything in hashtables/lists in RAM, in a single process, and serve out RPCs to client code (or run a webserver) as necessary. At this point, you don't have a useful product anyway, so it doesn't matter if it goes down and loses all data. Once you have something that's useful, work on moving state out to external systems: at this point, you have a pretty good idea what the schema should be, so you can save on a lot of migrations. Split into separate services when the computational resources exceed the ability of a single process to service them; before then, just split it up into libraries.

That's not a pure microservices approach, since you are sharing state (the database). It means that any development which changes or is impacted by a change in the DB has to be coordinated between teams.

The microservice approach is to have each service have its own database (MySQL) or schema (PostgreSQL).

Another solution is the use of a modular architecture, where separate modules are each responsible for their database tables' schema. Tables are not shared across modules; accessing data from a table requires calling its module's API.

Each module can be developed independently, and combined into one large service or several smaller services. It's helpful to be able to separate modules into their own service for load balancing and optimization, while having all modules in the same service eases administration of the system.

From previous experience, you start by lumping all modules together, then use monitoring software (i.e. ELK stack, collectd/Grafana) you decide where to split into services. Because the reality is you'll not really know where to split until you are running in production.

This is the approach of Django for example and works quite well for large projects.