Building a highly-available web service without a database

When I asked people working on it if they considered Redis or Mongo or Postgres with jsonb columns, they just said they considered all of those things but decided to roll out their own db anyway because "they understood it better".

This article gives off the same energy. I really hope it works out for you, but IMO spending innovation tokens to build a database is nuts.

https://www.microsoft.com/en-us/research/project/orleans-vir...

AWS and other cloud providers are money printers because a lot of engineers are insanely tied into established patterns of doing things and can't think through things at a fundamental level. Ive seen company backends where their entire AWS stacks could be replaced by a 2 EC2 instances behind a load balancer with a domain name, without affecting business flow.

We did something similar to the work in the OP post at my work, we had a bunch of ECS tasks for a service, where the service did another call to an upstream service to fetch some intermediate results. We wanted to cache results for lower response latency. People were working to set up a Redis cluster. Except the TPS of the service was like 0.1.

Took me one day to code a /sync api endpoint, which was just a replica of the main endpoint. The only difference is that the main endpoint would spin of a thread to call the /sync endpoint, whereas the /sync endpoint didn't. Both endpoints ended with caching the results in memory before returning. Easy as day, no additional infra costs necessary.

But overall, personally, I don't hate the "spending innovation tokens to build a database is nuts" sentiment too much, because it keeps me employed at high salary while doing minimal work, where things that really should be basic CS are considered innovation.

It claims to solve a bunch of problems by ignoring them. There are solid reasons why people distribute their applications across multiple machines. After reading this article I feel like we need to state a bunch of them.

Redundancy - what if one machine breaks either a hardware failure a software failure or a network failure (network partition where you can’t reach the machine or it can’t reach the internet)

Scaling- what if you can’t serve all of your customers from one machine ? Perhaps you have many customers and a small app or perhaps your app can use a lot of resources (maybe it loads gigs of data)

Deployment - what happens when we want to change the code and not go down if you are running multiple copies of your app you get this for cheap

There are tons of smaller benefits - right sizing your architecture What if the one machine you choose is not big enough you need to move to a new machine, with multiple machines you just increase the number of machines. You also get to use a variety of machine sizes and can choose ones that fit your needs so this flexibility allows you to choose cheaper machines

I feel like the authors don’t know why people invented the standard way of doing things.