If you’ve ever been in a situation where you do suddenly face scale and have to rip apart a legacy monolith that was built without scale in mind, you’ll chuckle at this article. It’s extremely painful.
[deleted]
Why is everything so scalable?
341–350 of 383 posts
Re: Why is everything so scalable?
#342Earlier quoted context omitted.
crud app for async chat app works only that far. when you start getting a lot of customers (companies, etc), big chat rooms, etc - things get complicated. i saw this kind of system that started as simple crud app, and many years later developers still try to resolve some of the originals sins.
If it takes about long to resolve "original sins" it wasn't simple enough to begin with.
"original sins" happened way before i joined the company. so i don't know the exact chronology, but app was started as very simple crud app. new features were added as client base increased. client base became more enterprisy. mobile apps + desktop apps appeared. new features were added to support them.
so something that started as simple node.js + nosql crud app written by not that experienced developers became over span of few years little shop of horror that was failing to scale without all kinds of crazy workarounds
Re: Why is everything so scalable?
#343The problem I never see addressed: scaling is not proportional to popularity. There is some sunk cost or "baseload" scale required to run a service.
Think google maps purchasing and processing satellite imagery; if you want to compete, you also need to regularly update petabytes of data. That's table stakes, even if you have one user.
Re: Why is everything so scalable?
#344Earlier quoted context omitted.
If you hire hammer experts then you're going to end up using a lot of hammers in your construction. The Azure experts aren't pitching Azure because they're trying to sell more Azure products. They do it because that's all they know and most likely because you don't know it so you'll be likely to come back to them for support when things inevitably need to evolve.
Also, the more you use your cloud vendor's various services in your code, the more subject you are to vendor lock-in. I won't name any names, but I'm pretty sure this is a big part of the reason why a specific cloud vendor pushed so very hard for us to push a bunch of data into their highly advanced NoSQL big data solution, when the data in question was perfectly happy continuing indefinitely to exist as a few tens o…
CosmosDB?
It’s okay, this is the Internet, you can name names.
Re: Why is everything so scalable?
#345Just to be honest for a bit here... we also should be asking what kind of scale? Quite a while ago, before containers were a thing at all, I did systems for some very large porn companies. They were doing streaming video at scale before most, and the only other people working on video at that scale were Youtube. The general setup for the largest players in that space was haproxy in front of nginx in front of several…
THANK YOU. People look at me like I’m insane when I tell them that their overly-complicated pipeline could be easily handled by a couple of beefy servers. Or at best, they’ll argue that “this way, they don’t have to manage infrastructure.” Except you do - you absolutely do. It’s just been partially abstracted away, and some parts like OS maintenance are handled (not that that was ever the difficult part of managing s…
[dead]
Re: Why is everything so scalable?
#346This piece is written with a pretty cliche dismissive tone that assumes that everything everyone else does is driven by cargo-culting if not outright ignorance. That people make these choices because they're just rushing to chase the latest trend. They're just trying to be cool, you see. Here's the thing, though: Almost every choice that leads to scalability also leads to reliability. These two patterns are effective…
> Almost every choice that leads to scalability also leads to reliability.
Yes, dealing with skew for every single change and hunting down bugs across network boundaries that could have been a function call is peak reliability.
Re: Why is everything so scalable?
#347Vertically scaling hetzner/do machines with some basic networking gets you farther than most startups survive.
Re: Why is everything so scalable?
#348Many startup business models have no chance of becoming profitable unless they reach a certain scale, but they might have less than 1% probability of reaching that scale. Making it scalable is easy work since it is deterministic, but growing customers is not. Another perspective is that the defacto purpose of startups (and projects at random companies) may actually be work experience and rehearsal for the day the fou…
> Making it scalable is easy work since it is deterministic
I don't think that necessarily follows. Especially the language choice is almost impossible to change - look at Facebook, Dropbox, etc. Facebook ended up creating an entirely new language that only they use, because it was impossible to rewrite in another language.
Language choice (and probably database choice too) are essentially locked in from the start, and they do affect scaling.
Growing customers is probably harder, but I don't buy "do everything in hacky Bash scripts because you can fix it later". Nor do I think having solid foundations means you need to be less agile. Would Dropbox have been less successful if they wrote their backend in Typescript? I doubt it.
Re: Why is everything so scalable?
#349Just to be honest for a bit here... we also should be asking what kind of scale? Quite a while ago, before containers were a thing at all, I did systems for some very large porn companies. They were doing streaming video at scale before most, and the only other people working on video at that scale were Youtube. The general setup for the largest players in that space was haproxy in front of nginx in front of several…
I have been advocating similar setups to this for years, no one wants to hear it. I get downright puzzled looks from people who think you need moon-landing monitorability of every little service and database...
We have 100k paying users and most of them don't actively use our service, yet we have microservice slop with k8 and load balancers galore running in big tech cloud provider and document databases.
From our whole setup I can only think of _maybe_ one thing that should be split into a separate dedicated service, everything else could happily live in a monolith backed by a relational database and a caching layer and a storage solution.
Re: Why is everything so scalable?
#350Earlier quoted context omitted.
Anyone that says, "they don’t have to manage infrastructure" I would invite them to deal with a multi-environment terraform setup and tell me again that about what they don't have to manage.
While terraform is not ideal it is much much more easy to deal with managed services in AWS than to deal with on premises baremetal servers. Most are biased because they like dealing with the kind of issues in on premises. They like dealing with the performance regressions, heat maps, kernel issues etc. Because why not? You are a developer and you need some way to exercise your skills. AWS takes that away and makes y…
> While terraform is not ideal it is much much more easy to deal with managed services in AWS than to deal with on premises baremetal servers.
If you have one or two baremetal servers it is not, but yes once you start having a lot of infra it is way better.
But you can get really, really far with one or two baremetal servers and some SaaS solutions...