From this thread, I feel a lot of people misunderstand what a scaleable system means for a scale-up startup or BigTechCo. It doesn’t mean cost-efficiency. It means the ability to solve scalability issues at 10x, 100x, 1000x load by throwing money at it (aka buying more instances/machines).
Yes, John Carmack is right that a bunch of scaleable systems that see reasonable load, with well-understood traffic requirements could be rewritten more efficient ways. But how long would it take? How much would the extra work cost? And what would happen if traffic goes up, to another 10x, 100x? Can you throw still throw money at that problem, with this new and efficient system already in-place?
One of my memorable stories comes from an engineer I worked with when we rewrote our payments system[1]. He told me how at a previous payments company, they had a system that was written in this kind of efficient way, and needed to run on one machine. As the scale went up, the company kept buying bigger hardware, at one point buying the largest commercially available mainframe (we’re talking the cost being in the multi-millions for the hardware). But the growth was faster and they couldn’t keep up. Downtime after downtime followed at peak times, making hundreds of thousands of losses per downtime.
They split into two teams. Team #1 kept making performance tweaks on the existing hardware to try to get performance wins, and increase reliability during peak load. Team #2 rewrote their system in a vertically scaleable way. Team #1 struggled and the outages kept getting worse. Team #2 delivered a new system quickly and the company transitioned their system over.
What they gained was not cost savings: it was the ability to (finally!) throw money at their growth problems. Now, when traffic went up, they could commission new machines and scale with traffic. And they could eventually throw away their mainframe.
Years later, they started to optimise the performance of the system, making millions of $ in savings. The new system - like the old one - cost millions to run. But that was besides the point. Finally, they stopped bleeding tens of millions in lost revenue per year due to their inability to handle sudden, high load.
It’s all about trade-offs. Is cost your #1 priority and are you okay spending a lot of development time on optimising your system? Go with C++, Erlang or some other, similarly efficient language. Is product-market fit more important, with the ability to have high uptime, that you can buy? Use the classic, horizontally scaleable distributed systems stack, and worry about optimising later, when you have stable traffic and optimisation is more profitable over product development.
[1] https://blog.pragmaticengineer.com/distributed-architecture-...