I've seen my share of insanely over-engineered Azure locked-in applications that could easily have been run on an open source stack on a $20 VM.
But what if payroll grows to 100M internal users?
Why is everything so scalable?
141–150 of 383 posts
Re: Why is everything so scalable?
#142Everything is scalable, because it became very easy to write scalable software. I guess that's the reason.
Re: Why is everything so scalable?
#143Just to be honest for a bit here... we also should be asking what kind of scale? Quite a while ago, before containers were a thing at all, I did systems for some very large porn companies. They were doing streaming video at scale before most, and the only other people working on video at that scale were Youtube. The general setup for the largest players in that space was haproxy in front of nginx in front of several…
> The general setup for the largest players in that space was haproxy in front of nginx in front of several PHP servers in front of a MySQL database that had one primary r/w with one read only replica. You'd be surprised that the most stable setups today are run this way. The problem is that this way it's hard to attract investors; they'll assume you are running on old or outdated tech. Everything should be serverles…
> The problem is that this way it's hard to attract investors; they'll assume you are running on old or outdated tech. Everything should be serverless, agentic and, at least on paper, hyperscalable, because that sells further.
Investors don't give a shit about your stack
Re: Why is everything so scalable?
#144I have read and watched these articles and videos where people seem to have a problem with Microservice, Kubernetes, cloud providers, or anything that's not a PHP server sitting behind an nginx running on a $5 VPS. I have also seen the front-end analogy of these types of posts, where anything that is not written using HTML, CSS, and jQuery is unnecessary bloat. I will soon write a blog, which I think will cover more…
I thought capitalism was about adding value, not conflict of interest
Re: Why is everything so scalable?
#145Earlier quoted context omitted.
I've seen similar with Firebase. Luckily I took over as tech lead at this company, so I was able to migrate us to Postgres. Amusingly, as well as being more reliable, the Postgres version (on a single small database instance) was also much faster than the previous Firebase-based version (due to it enabling JOINs in the database rather than in application code).
Funnily enough prior to this startup I had worked at a rainforest-themed big tech co where we ran all kinds of stuff on MySQL without issue, at scales that dwarfed what this startup was up to by 3-4 orders of magnitude. I feel like that's kind of the other arm of this whole argument: on the one hand, you ain't gonna need that "scalable" thing. On the other hand, the "unscalable" thing scales waaaaaay higher than you…
> On the other hand, the "unscalable" thing scales waaaaaay higher than you are led to believe.
Agreeing with you... Any reasonable database will scale pretty far if you put in a machine with 160 cores and 3 TB of RAM. And that's just a single socket board.
There's no reason to do anything other than get bigger machines until you're near or at the limits of single socket. Dual socket and cpu generations should cover you for long enough to move to something else if you need to. Sharding a traditional database works pretty well in a lot of cases, and it mostly feels like the regular database.
Re: Why is everything so scalable?
#146Earlier quoted context omitted.
Funnily enough prior to this startup I had worked at a rainforest-themed big tech co where we ran all kinds of stuff on MySQL without issue, at scales that dwarfed what this startup was up to by 3-4 orders of magnitude. I feel like that's kind of the other arm of this whole argument: on the one hand, you ain't gonna need that "scalable" thing. On the other hand, the "unscalable" thing scales waaaaaay higher than you…
> On the other hand, the "unscalable" thing scales waaaaaay higher than you are led to believe. Agreeing with you... Any reasonable database will scale pretty far if you put in a machine with 160 cores and 3 TB of RAM. And that's just a single socket board. There's no reason to do anything other than get bigger machines until you're near or at the limits of single socket. Dual socket and cpu generations should cover…
That, and a lot of companies don't have the scale they think they have.
The Postgres database for a company I worked for (that was very concerned about scaling when they interviewed me because their inefficient "nosql" solution was slow) ran very happily on a machine with 2 shared CPU cores and 4GB RAM.
Re: Why is everything so scalable?
#147Earlier quoted context omitted.
THANK YOU. People look at me like I’m insane when I tell them that their overly-complicated pipeline could be easily handled by a couple of beefy servers. Or at best, they’ll argue that “this way, they don’t have to manage infrastructure.” Except you do - you absolutely do. It’s just been partially abstracted away, and some parts like OS maintenance are handled (not that that was ever the difficult part of managing s…
Depending on your regulatory environment, it can be cost-effective to not have to maintain your own data center with 24/7 security response, environmental monitoring, fire suppression systems, etc. (of course, the majority of businesses are probably not interested in things like SOC 2)
The only companies directly dealing with that type of stuff are the ones already at such a scale where they need to actually build their own data centers. Everyone else is just renting space somewhere that already takes care of those things and you just need to review their ISO/SOC reports.
This kind of argument comes from the cloud provider marketing playbook, not reality.
Re: Why is everything so scalable?
#148Re: Why is everything so scalable?
#149Earlier quoted context omitted.
THANK YOU. People look at me like I’m insane when I tell them that their overly-complicated pipeline could be easily handled by a couple of beefy servers. Or at best, they’ll argue that “this way, they don’t have to manage infrastructure.” Except you do - you absolutely do. It’s just been partially abstracted away, and some parts like OS maintenance are handled (not that that was ever the difficult part of managing s…
Anyone that says, "they don’t have to manage infrastructure" I would invite them to deal with a multi-environment terraform setup and tell me again that about what they don't have to manage.
Those are the ones that also usually tell you you can just stitch together a few SaaS products and it's magic.
Re: Why is everything so scalable?
#150Earlier quoted context omitted.
THANK YOU. People look at me like I’m insane when I tell them that their overly-complicated pipeline could be easily handled by a couple of beefy servers. Or at best, they’ll argue that “this way, they don’t have to manage infrastructure.” Except you do - you absolutely do. It’s just been partially abstracted away, and some parts like OS maintenance are handled (not that that was ever the difficult part of managing s…
Working on various teams operating on infrastructure that ranged from a rack in the back of the office, a few beefy servers in a colo, a fleet of Chef-managed VMs, GKE, ECS, and various PaaSes, what I've liked the most about the cloud and containerized workflows is that they wind up being a forcing function for reproducibility, at least to a degree. While it's absolutely 100% possible to have a "big beefy server arch…
You have to remove admin rights to your admins then, because scrappy enough DevOps/platform engineers/whatever will totally hand-edit your AWS infra or Kubernetes deployments. I suffered that first hand. And it's even worse that in the old days, because at least back in the day it was expected.