Live data from Hacker News

Why is everything so scalable?

stavros.io

71–80 of 383 posts

Re: Why is everything so scalable?

#71
post #54

Comes down to knowing when to stop. You don’t really want to DIY your own orchestrator etc. So better off just using kubernetes. But then not going too far down that rabbit hole. ie yes kubernetes but the simplest vanilla version of it you can manage

I wouldn’t start with K8s, and I’ve administered it at multiple companies. Unless every one of your initial hires is a SWE turned SRE, you’re in for a bad time (and you don’t need it).

I’d personally start with Linux services on some VMs, but Docker Compose is also valid. There are plenty of wrappers around Compose to add features if you’d like.

Re: Why is everything so scalable?

#72

Hilariously written but also too true. One start up I worked at we had 2 Kubernetes clusters and a rat's nest of microservices for an internal tool that, had we been actually successful at delivering sufficient value would have been used by at most a 100 employees (and those would unlikely be concurrent). And this was an extremely highly valued company at the time. Another place I worked at we were paying for 2 dev o…

What were these dev ops engineers doing all day? Surely you can only polish a cluster so much before it's done and there is nothing left to do?

Re: Why is everything so scalable?

#74
post #41

I've seen startups killed because of one or two "influential" programmers deciding they need to start architecturing the project for 1000TPS and 10K daily users, as "that's the proper way to build scalable software", while the project itself hasn't even found product-market fit yet and barely has users. Inevitably, the project needs to make a drastic change which now is so painful to do because it no longer fits the…

I've seen senior engineers get fired and the business suffer a setback because they didn't have any way to scale beyond a single low spec VPS from a budget provider, and their system crashed when a hall full of students tried to sign up together during a demo and each triggered 200ms of bcrypt CPU activity.

This seems weird. I have a lot of experience with rails which is considered super slow. But the scenario you describe is trivial. Just get a bigger VPS and change a single env var. even if you fucked up everything else like file storage etc you can still to that. If you build your whole application in way where you can’t scale anything you should be fired. That is not even that easy

Re: Why is everything so scalable?

#75
post #52
post #44

Earlier quoted context omitted.

The architecture you describe is ok because in the end it is a fairly simple website. Little user interaction, limited amount of content (at most a few million records), few content changes per day. The most complex part is probably to have some kind of search engine but even with 10 million videos an ElasticSearch index is probably no larger than 1GB. The only problem is that there is a lot of video data.

This is probably also true for 98% of startups. I think most people don't realise that "10 million" records is small, for a computer. (That said, I have had to deal with code that included an O(n^2) de-duplication where the test data had n ~= 20,000, causing app startup to take 20 minutes; the other developer insisted there was no possible way to speed this up, later that day I found the problem, asked the CTO if the…

I thought you were going to say to reduced O(n^2) to O(n*log(n)), but you just deleted the operation. Normally I'd say that's great, but just how much duplicate data is being left around now? Is that OK?

Re: Why is everything so scalable?

#76
post #70
post #29

Just to be honest for a bit here... we also should be asking what kind of scale? Quite a while ago, before containers were a thing at all, I did systems for some very large porn companies. They were doing streaming video at scale before most, and the only other people working on video at that scale were Youtube. The general setup for the largest players in that space was haproxy in front of nginx in front of several…

Are those over engineered systems even actually scalable? I know teams who designed a CQRS architecture using messages queues and a distributed NoSQL database and fail to sustain 10req/s for a read in something that is basically a CRUD application. Heck once someone literally said "But we use Kafka, why aren't we fast?!".

Exactly this, every time I see kafka or similar its a web of 10M microprocesses that take more time in invocation alone than if you just ran the program in one go.

Re: Why is everything so scalable?

#77

If you’ve ever been in a situation where you do suddenly face scale and have to rip apart a legacy monolith that was built without scale in mind, you’ll chuckle at this article. It’s extremely painful.

Legitimately asking, how? The only bottleneck should be the DB, and if you can saturate a 128-core DB, I want to see your queries and working set size. Not saying it can’t happen, but it’s rare that someone has actually maxed out MySQL or Postgres without there being some serious schema and query flaws, or just poor / absent tuning.

Re: Why is everything so scalable?

#78
post #72

Hilariously written but also too true. One start up I worked at we had 2 Kubernetes clusters and a rat's nest of microservices for an internal tool that, had we been actually successful at delivering sufficient value would have been used by at most a 100 employees (and those would unlikely be concurrent). And this was an extremely highly valued company at the time. Another place I worked at we were paying for 2 dev o…

What were these dev ops engineers doing all day? Surely you can only polish a cluster so much before it's done and there is nothing left to do?

You should have seen the architecture they came up with... it had ALL the bells and whistles you could possibly imagine and cost an absolute fortune.

Of course they eventually got bored and quit. And then it became really annoying since no one else understood anything about it.

Re: Why is everything so scalable?

#79

I read this, and have the opposite experience. Your monolith will fester as developers step on each others toes. You aren’t solving for scalability, you’re solving for sovereignty. Giving other teams the ability to develop their own service without needing to conform to your archaic grey beard architecture restrictions and your lack of understanding what a pod is or how to get your logs from your cloud. No, this whol…

>step on each others toes

Which leads us to a huge problem I’ve seen over the past few decades.

Too many developers for the task at hand. It’s easier for large companies to hire 100 developers with a lower bar that may or may not be a great fit than it is to hire 5 experts.

Then you have a 100 developers that you need to keep busy and not all of them can be busy 100% of the time because most people aren’t good at making their own impactful work. Then instead of trying to actually find naturally separate projects for some of them to do, you attempt to artificially break up your existing project in a way that 100 developers can work on together (and enforce those boundaries at through a network).

This artificial separation fixes some issues (merge conflicts, some deployment issues), but it causes others (everything is a distributed system now, multi stage and multi system deployments required for the smallest changes, massive infrastructure, added network latency everywhere).

That’s not to say that some problems aren’t really so big that you need a huge number of devs, but the vast majority aren’t.

> they don’t have to reinvent the wheel

Everything is a trade off, but we shouldn’t discount the cost of using generic solutions in place of bespoke ones.

Generic solutions are never going to be as good of a fit as something designed to do exactly what you need. Sometimes the tradeoff is worth it. Sometimes it’s isn’t. Like when you need to horizontally scale just to handle the overhead. Or when you have to maintain a fork of a complex system that does way more than you need.

It’s the same problem as hiring 100 generic devs instead of 5 experts. Sometimes worth it. Sometimes not.

There’s another issue here too. If not enough people are reinventing the wheel we get stuck in local optima.

The worst part is that not enough people spend enough time even thinking about these issues to make informed decisions regarding the tradeoffs they are making.

Re: Why is everything so scalable?

#80

> you usually only have one database What if I use the cloud? I don't even know how many servers my database runs on. Nor do I care. It's liberating not having to think about it at all.

And your cloud providers thank you for giving their executives a third yacht.
Post reply on HN