Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is everything so scalable?

stavros.io

Post reply on HN
Print view View original

251–260 of 383 posts

Re: Why is everything so scalable?

#251

Post by hedora »

Groxx wrote: post #197
hedora wrote: post #135

Earlier quoted context omitted.

I've notice a strong correlation between artificially over-distributing, and not understanding things like the CAP theorem. So, you end up with a slow system that's added a bunch of unsolvable distributed systems problems on its fast path. (Most distributed systems problems are solvable, but only if the person that architected the system knows what they're doing. If they know what they're doing, they won't over-distr…

You can solve just about any distributed systems problem by accepting latency, but nobody wants to accept latency :) ...despite the vast majority of latency issues being extremely low-hanging fruit, like "maybe don't have tens of megabytes of data required to do first paint on your website" or "hey maybe have an index in that database?".

Well, yeah, but the people that create the issues typically solve them by just corrupting the crap out of app state and adding manual ops procedures.

Re: Why is everything so scalable?

#252

Post by dwaltrip »

esafak wrote: post #228
strken wrote: post #41

Earlier quoted context omitted.

I've seen senior engineers get fired and the business suffer a setback because they didn't have any way to scale beyond a single low spec VPS from a budget provider, and their system crashed when a hall full of students tried to sign up together during a demo and each triggered 200ms of bcrypt CPU activity.

I will never forget the time my university's home-grown Web-based registration system crashed at the beginning of the semester, and the entirety of the university's student body had to form a line in order to have their registration entered manually. I waited a whole day, and they did not get round to me by night, so I had to wait the next day too.

“Knowing what’s reasonable” matters.

If you have a product that’s being deployed for a new school year, yeah you should be prepared for any one-time load for that time period.

Many products don’t have the “school year just started” spikes. But some do.

It requires careful thought, pragmatism, and business sense to balance everything and achieve the most with the available resources.

Re: Why is everything so scalable?

#253

Post by moffkalast »

pja wrote: post #122
otabdeveloper4 wrote: post #39

Earlier quoted context omitted.

> 1000TPS and 10K daily users That is not a lot. You can host that on a Raspberry Pi.

Not if you’re going to be “web scale” (tm) you can’t.

You put one Mongo shard on each Pi, they are the secret ingredient in the web scale sauce.

Re: Why is everything so scalable?

#254

Post by vidarh »

jkaplowitz wrote: post #176
darkwater wrote: post #150

Earlier quoted context omitted.

You have to remove admin rights to your admins then, because scrappy enough DevOps/platform engineers/whatever will totally hand-edit your AWS infra or Kubernetes deployments. I suffered that first hand. And it's even worse that in the old days, because at least back in the day it was expected.

Or at least you have to automatically destroy and recreate all nodes / VMs / similar every N days, so that nobody can pretend that any truly unavoidable hand-edits during emergency situations will persist. Possibly also control access to the ability to do hand edits behind a break-glass feature that also notifies executives or schedules a postmortem meeting about why it was necessary to do that.

I know of at least one organisation that'd automatically wipe every instance on (ssh-)user logout, so you could log in to debug, but nothing you did would persist at all. I quite like that idea, though sometimes being able to e.g. delay the wipe for up to X hours might be slightly easier to deal with for genuinely critical emergency fixes.

But, yes, gating it behind notifications would also be great.

Re: Why is everything so scalable?

#255

Post by treis »

yoyohello13 wrote: post #249
vidarh wrote: post #241

Earlier quoted context omitted.

I do consulting in this space, and I'm torn: I make much more money managing infrastructure from clients who insist on AWS. But it's much more enjoyable to work with people who knows how to keep it simple.

I worked on a project for my company (a low volume basic web app) and I suggested we could just start the whole thing on one server. They brought on some Azure consultants and the project ballooned out to months of work and all kinds of services. I’m convinced most of the consultants were just piling on services so they could make more money.

I joke with my boss that all our shit ends up running on a single server in some Amazon data center. It's probably not true but if you add up everything we do it's pretty close to one big server.

Re: Why is everything so scalable?

#256

Post by alexchantavy »

BirAdam wrote: post #29

Just to be honest for a bit here... we also should be asking what kind of scale? Quite a while ago, before containers were a thing at all, I did systems for some very large porn companies. They were doing streaming video at scale before most, and the only other people working on video at that scale were Youtube. The general setup for the largest players in that space was haproxy in front of nginx in front of several…

Around 5 years ago the metagame was to make everything horizontally scalable.

Now it seems things are swinging back the other direction and articles like "Use One Big Server" are getting re-discussed: https://news.ycombinator.com/item?id=45085029

Re: Why is everything so scalable?

#257

Post by vidarh »

tracker1 wrote: post #239
macNchz wrote: post #130

Earlier quoted context omitted.

Working on various teams operating on infrastructure that ranged from a rack in the back of the office, a few beefy servers in a colo, a fleet of Chef-managed VMs, GKE, ECS, and various PaaSes, what I've liked the most about the cloud and containerized workflows is that they wind up being a forcing function for reproducibility, at least to a degree. While it's absolutely 100% possible to have a "big beefy server arch…

I'm still a pretty big fan of Docker (compose) behind Caddy as a reverse-proxy... I think that containers do offer a lot in terms of application support... even if it's a slightly bigger hoop to get started with in some ways.

I'm working on an app server that's auto deploying itself behind Caddy + DNS/SSL aut config. Caddy is amazing, and there really should be no reason for complex setups for most people these days... I've worked on some huge systems, but most systems can run in trivially simple setups given modern hardware.

Re: Why is everything so scalable?

#258

Post by ericwood »

hedora wrote: post #115
nasmorn wrote: post #74

Earlier quoted context omitted.

This seems weird. I have a lot of experience with rails which is considered super slow. But the scenario you describe is trivial. Just get a bigger VPS and change a single env var. even if you fucked up everything else like file storage etc you can still to that. If you build your whole application in way where you can’t scale anything you should be fired. That is not even that easy

People screw up the bcrypt thing all the time. Pick a single threaded server stack (and run on one core, because Kubernetes), then configure bcrypt so brute forcing 8 character passwords is slow on an A100. Configure kubernetes to run on a medium range CPU because you have no load. Finally, leave your cloud provider's HTTP proxy's timeout set to default. The result is 100% of auth requests timeout once the login queu…

Fond memories of a job circa 2013 on a very large Rails app where CI times were sped up by a factor of 10 when someone realized bcrypt was misconfigured when running tests and slowing things down every time a user was created through a factory.

Re: Why is everything so scalable?

#259

Post by vidarh »

huflungdung wrote: post #94
sgarland wrote: post #47

Earlier quoted context omitted.

THANK YOU. People look at me like I’m insane when I tell them that their overly-complicated pipeline could be easily handled by a couple of beefy servers. Or at best, they’ll argue that “this way, they don’t have to manage infrastructure.” Except you do - you absolutely do. It’s just been partially abstracted away, and some parts like OS maintenance are handled (not that that was ever the difficult part of managing s…

What I say is that we massively underestimate just how fast computers are these days

Most younger devs just have no concept on how limited hardware we ran services on...

I used to run a webmail system with 2m accounts on hardware with less total capacity (ram, disk, CPU throughput) than my laptop...

What's more: It was a CGI (so new process for every request), and the storage backend spawned separate processes per user.

Re: Why is everything so scalable?

#260

Post by jandrese »

ahartmetz wrote: post #190
iknowstuff wrote: post #165

Earlier quoted context omitted.

https://us.ovhcloud.com/bare-metal/prices/?display=list also pretty sure 24 cores is like 48 cloud “cores” which are usually just hyper threads right?

IME, a cloud "core" is even worse than a hyperthread. I'm not sure if they oversubscribe, or underclock, or if it's virtualization overhead... but anyway, not great.

They oversubscribe.
Post reply on HN