Scale articles are too focused on architecture. What about business problems that come with scale. At a certain scale rare events are common many cases cease to be fixable by some random process that involves humans you have to handle a lot more business scenarios with your code.
Why is everything so scalable?
51–60 of 383 posts
Re: Why is everything so scalable?
#52Just to be honest for a bit here... we also should be asking what kind of scale? Quite a while ago, before containers were a thing at all, I did systems for some very large porn companies. They were doing streaming video at scale before most, and the only other people working on video at that scale were Youtube. The general setup for the largest players in that space was haproxy in front of nginx in front of several…
The architecture you describe is ok because in the end it is a fairly simple website. Little user interaction, limited amount of content (at most a few million records), few content changes per day. The most complex part is probably to have some kind of search engine but even with 10 million videos an ElasticSearch index is probably no larger than 1GB. The only problem is that there is a lot of video data.
This is probably also true for 98% of startups.
I think most people don't realise that "10 million" records is small, for a computer.
(That said, I have had to deal with code that included an O(n^2) de-duplication where the test data had n ~= 20,000, causing app startup to take 20 minutes; the other developer insisted there was no possible way to speed this up, later that day I found the problem, asked the CTO if there was a business reason for that de-duplication, removed the de-duplication, and the following morning's stand-up was "you know that 20 minute startup you said couldn't possibly be sped up? Yeah, well, I sped it up and now it takes 200ms")
Re: Why is everything so scalable?
#53I read this, and have the opposite experience. Your monolith will fester as developers step on each others toes. You aren’t solving for scalability, you’re solving for sovereignty. Giving other teams the ability to develop their own service without needing to conform to your archaic grey beard architecture restrictions and your lack of understanding what a pod is or how to get your logs from your cloud.
No, this whole article reads like someone who is crying that they no longer have their AS/200. Bye. The reason people use AWS and all those 3rd party is so they don’t have to reinvent the wheel which this author seems hell bent on.
Why are we using TCP when a Unix file is fine… why are we using databases when a directory and files is fine? Why are we scaling when we aren’t Google when my single machine can serve a webpage? Why am I getting paid to be an engineer while eschewing all the things that we have advanced over the last two decades?
Yeah, these are not the right questions. The real question should be: “Now that we have scale what are we gonna do with it?”
Re: Why is everything so scalable?
#54Comes down to knowing when to stop. You don’t really want to DIY your own orchestrator etc. So better off just using kubernetes. But then not going too far down that rabbit hole.
ie yes kubernetes but the simplest vanilla version of it you can manage
Re: Why is everything so scalable?
#55Frontend folks weren’t happy with how simple things were, so after seeing microservices, they invented microfrontends, and balance in complexity restored.
Re: Why is everything so scalable?
#56I've seen startups killed because of one or two "influential" programmers deciding they need to start architecturing the project for 1000TPS and 10K daily users, as "that's the proper way to build scalable software", while the project itself hasn't even found product-market fit yet and barely has users. Inevitably, the project needs to make a drastic change which now is so painful to do because it no longer fits the…
I've seen senior engineers get fired and the business suffer a setback because they didn't have any way to scale beyond a single low spec VPS from a budget provider, and their system crashed when a hall full of students tried to sign up together during a demo and each triggered 200ms of bcrypt CPU activity.
That’s a skill issue, not an indictment on the limitations of the architecture. You can spin up N servers and load-balance them, as TFA points out. If the server is a snowflake and has nothing in IaC, again, not an architectural issue, but a personnel / knowledge issue.
Re: Why is everything so scalable?
#57I've seen startups killed because of one or two "influential" programmers deciding they need to start architecturing the project for 1000TPS and 10K daily users, as "that's the proper way to build scalable software", while the project itself hasn't even found product-market fit yet and barely has users. Inevitably, the project needs to make a drastic change which now is so painful to do because it no longer fits the…
I've seen senior engineers get fired and the business suffer a setback because they didn't have any way to scale beyond a single low spec VPS from a budget provider, and their system crashed when a hall full of students tried to sign up together during a demo and each triggered 200ms of bcrypt CPU activity.
I frankly don't believe that in a workplace where an userbase can be characterized as a "hall full of students" anyone was fired overnight.
Doesn't happen at these places.
Reprimanded, maybe
Re: Why is everything so scalable?
#58> The first problem every startup should solve is “how do we have enough money to not go bust in two months”, but that’s a hard problem, whereas scalability is trivially solvable by reading a few engineering blogs [...] Do you know what the difference between Google and your startup is? It’s definitely not scalability, you’ve solved that problem. It’s that Google has billions upon billions with which to pay for that scalability, which is really good because scalability is expensive.
Too true. Now that I've stepped into an "engineering leadership" role and spend as much time looking at finances as I do at code, I've formed the opinion that in 99.999% of cases, engineering problems are really business problems. If you could throw infinite time and money at the technical challenges, they'd no longer be challenging. But businesses, especially startups, don't have infinite (or even "some") money and time, so the challenge is doing the best engineering work you can, given time and budget constraints.
> The downsides [of the monolith approach]
I like the article's suggestion of using explicitly defined API boundaries between modules, and that's a good approach for a monolith. However one massive downside that cannot be ignored -- by having a single monolith you now have an implicit dependency on the same runtime working on all parts of your code. What I mean by this is, all your code is going to share the same Python version and same libraries (particularly true in Python, where it's not a common/well-supported use case to have multiple versions of library dependencies). This means that if you're working on Module A, and you realize you need a new feature from Pandas 2.x, but the rest of the code is on Pandas 1.x... well, you can't upgrade unless you go and fix Modules B, C, D ... Z to work with Pandas 2.
This won't be an issue at the start, but it's worth pointing out. Being forced to upgrade a core library or language runtime and finding out it's a multi-month disruptive project can be brutal.
Re: Why is everything so scalable?
#59Hilariously written but also too true. One start up I worked at we had 2 Kubernetes clusters and a rat's nest of microservices for an internal tool that, had we been actually successful at delivering sufficient value would have been used by at most a 100 employees (and those would unlikely be concurrent). And this was an extremely highly valued company at the time. Another place I worked at we were paying for 2 dev o…
This sounds just about right: I have read that Kubernetes is the greek term for "more containers than customers".
Re: Why is everything so scalable?
#60Earlier quoted context omitted.
> Almost every choice that leads to scalability also leads to reliability. Empirically, that does not seem to be the case. Large scalable systems also go offline for hours at a time. There are so many more potential points of failure due to the complexity. And even with a single regular server, it's very easy to keep a live replica backup of the database and point to that if the main one goes down. Which is a common…
>Empirically, that does not seem to be the case. Failures are astonishingly, vanishingly rare. Like it's amazing at this point how reliable almost every system is. There are a tiny number of failures at enormous scale operations (almost always due to network misconfigurations, FWIW), but in the grand scheme of things we've architected an outrageously reliable set of platforms. >That's not scaling, just redundancy. In…
> Failures are astonishingly, vanishingly rare
You and I must be using different sites and different clouds.
There's a reason isitdownrightnow.com exists. And why HN'ers are always complaining about service status pages being hosted on the same services.
By your logic, AWS and Azure should fail once in a millennium, yet they regularly bring down large chunks of the internet.
Literally last week: https://cyberpress.org/microsoft-azure-faces-global-outage-i...