Live data from Hacker News

Why is everything so scalable?

stavros.io

291–300 of 383 posts

Re: Why is everything so scalable?

#291
post #227

Earlier quoted context omitted.

While terraform is not ideal it is much much more easy to deal with managed services in AWS than to deal with on premises baremetal servers. Most are biased because they like dealing with the kind of issues in on premises. They like dealing with the performance regressions, heat maps, kernel issues etc. Because why not? You are a developer and you need some way to exercise your skills. AWS takes that away and makes y…

> performance regressions, heat maps, kernel issues etc. > AWS takes that away and makes you focus on the product. ha ha ha no. Have been dealing with kernel issues on my AWS machines for a long time. They lock up under certain kinds of high load. AWS support is useless. Experimenting with kernel version leads to performance regressions. AWS is great if your IT/purchasing department is inefficient. Getting a new AWS…

>AWS is great if your IT/purchasing department is inefficient

Fwiw, I think a lot of companies have this problem.

Re: Why is everything so scalable?

#292
post #29

Just to be honest for a bit here... we also should be asking what kind of scale? Quite a while ago, before containers were a thing at all, I did systems for some very large porn companies. They were doing streaming video at scale before most, and the only other people working on video at that scale were Youtube. The general setup for the largest players in that space was haproxy in front of nginx in front of several…

THANK YOU. People look at me like I’m insane when I tell them that their overly-complicated pipeline could be easily handled by a couple of beefy servers. Or at best, they’ll argue that “this way, they don’t have to manage infrastructure.” Except you do - you absolutely do. It’s just been partially abstracted away, and some parts like OS maintenance are handled (not that that was ever the difficult part of managing s…

On the other hand, I know a lot of people who spend more time / salary messing around with their infra than the couple hundred bucks they've saved from not pressing a couple of buttons on vercel / cloudfare

There's a time and place for just deploying quickly to a cloud provider versus trying to manage your infra. It's a nuanced tradeoff that rarely has a clear winner.

Re: Why is everything so scalable?

#293

Hilariously written but also too true. One start up I worked at we had 2 Kubernetes clusters and a rat's nest of microservices for an internal tool that, had we been actually successful at delivering sufficient value would have been used by at most a 100 employees (and those would unlikely be concurrent). And this was an extremely highly valued company at the time. Another place I worked at we were paying for 2 dev o…

I work at a place with 8 k8s clusters. We needed to evolve from generation 2 to generation 3 because of "manageability" or something. Gen 3 needed two clusters instead of one. Now we have 8 * (1 + 2) = 24 clusters.

Happy days.

Re: Why is everything so scalable?

#294
post #229

Earlier quoted context omitted.

Could you explain this some more? How are your costs so low in comparison? Are you using serverless?

That's how the cloudy platforms get you. They're very cheap on the low end, until they're not.

I've heard this a few times. Can you explain a bit more why you think that's a problem.

I've always made the assumption that once they become "not cheap" you now have the cost to offset investment against.

Re: Why is everything so scalable?

#295
post #72

Hilariously written but also too true. One start up I worked at we had 2 Kubernetes clusters and a rat's nest of microservices for an internal tool that, had we been actually successful at delivering sufficient value would have been used by at most a 100 employees (and those would unlikely be concurrent). And this was an extremely highly valued company at the time. Another place I worked at we were paying for 2 dev o…

What were these dev ops engineers doing all day? Surely you can only polish a cluster so much before it's done and there is nothing left to do?

It takes approximately 3 months to get it "just right". Luckily, k8s releases (and CNI and auth sidecars, and...) release every 2.8 months.

Re: Why is everything so scalable?

#296
post #229
post #163

Actually, scalability is cheap. Our AWS bill until recently was around $160-$200 a month. To get the level of HA and performance would require at least 20 boxes in two data centers. Dev/test/prod with an HA db and a backend that never dies. I’ve built those on bare iron and they’re expensive. If you’re going for saas and customers that don’t care about your infrastructure then a hetzner box is fine. But really, creat…

Could you explain this some more? How are your costs so low in comparison? Are you using serverless?

It depends on you understanding your app and how things need to be structured. We have what essentially is a video CMS, so we have two parts: a management UI that end-users use and a backend that actually delivers the video and collects metrics.

They are essentially two products, and are designed that way; if the management UI barfed the backend would continue along forever.

You can combine management and delivery in one app, but that makes delivery more fragile and will be slower because presumably it has to invoke a lot of useless stuff just to deliver bytes. I remember working with a spring app that essentially built and destroyed the whole spring runtime just to serve a request, which was an unbelievably dumb thing to do. Spring became the bottleneck, and for most requests there was actually no work done; 99% of the time was in spring doing spring things.

So really, once you separate the delivery and management it becomes easier to figure out the minimum amount of stuff you need. Redis, because you need to cache a bunch of metadata and handle lots of connections. Mysql, because you need a persistent store. Lambda, as a thin layer between everything. And a CDN, because you don't want to serve stuff out of AWS if you can help it. SQS for what essentially becomes job control. And for metric collection we use fastly with synthetic logging.

To be fair, our AWS cost was low but our CDN cost is like $1800/mo for some number of PB/mo (5? 10? I forget).

In the old days this would require at least (2 DB + 2 App server + 2 NAS) * 2 locations = 8 boxes. If we were going to do the networking ourselves we'd add 4 f5s. Ideally we'd have the app server, redis, and the various lambdas on different boxes, so 2 redis + 2 runners = 8 more servers. If we didn't use f5s we'd have 2 reverse proxies as the front end at each location. Each box would have 2 PSUs, at least a raid 1, dual NICs, and ECC. I think the lowest end Dell boxes with those features are like $5k each? Today I'd probably just stuff some 1TB SSDs in them and mirror them instead of going SAS. The NAS would be hard to spec because you have to figure out how much storage you need and they can be a pain to reconfigure. You don't want to spend too much up front, but you also don't want to have downtime while you add some more drive space.

Having built this out, it's not as easy as you'd think. I've been lucky enough to have built this sort of thing a few times. It's fun to do, but maintaining it can be a PITA. If you don't believe in documentation your deployment will fail miserably because you did something out of order.

Re: Why is everything so scalable?

#297
post #70
post #29

Just to be honest for a bit here... we also should be asking what kind of scale? Quite a while ago, before containers were a thing at all, I did systems for some very large porn companies. They were doing streaming video at scale before most, and the only other people working on video at that scale were Youtube. The general setup for the largest players in that space was haproxy in front of nginx in front of several…

Are those over engineered systems even actually scalable? I know teams who designed a CQRS architecture using messages queues and a distributed NoSQL database and fail to sustain 10req/s for a read in something that is basically a CRUD application. Heck once someone literally said "But we use Kafka, why aren't we fast?!".

Eh, they scale between $1000 and $10000 per month fairly easily. I’m not sure about the requests though.

Re: Why is everything so scalable?

#298
post #241

Earlier quoted context omitted.

I do consulting in this space, and I'm torn: I make much more money managing infrastructure from clients who insist on AWS. But it's much more enjoyable to work with people who knows how to keep it simple.

I worked on a project for my company (a low volume basic web app) and I suggested we could just start the whole thing on one server. They brought on some Azure consultants and the project ballooned out to months of work and all kinds of services. I’m convinced most of the consultants were just piling on services so they could make more money.

If you hire hammer experts then you're going to end up using a lot of hammers in your construction. The Azure experts aren't pitching Azure because they're trying to sell more Azure products. They do it because that's all they know and most likely because you don't know it so you'll be likely to come back to them for support when things inevitably need to evolve.

Re: Why is everything so scalable?

#299
post #115
post #74

Earlier quoted context omitted.

This seems weird. I have a lot of experience with rails which is considered super slow. But the scenario you describe is trivial. Just get a bigger VPS and change a single env var. even if you fucked up everything else like file storage etc you can still to that. If you build your whole application in way where you can’t scale anything you should be fired. That is not even that easy

People screw up the bcrypt thing all the time. Pick a single threaded server stack (and run on one core, because Kubernetes), then configure bcrypt so brute forcing 8 character passwords is slow on an A100. Configure kubernetes to run on a medium range CPU because you have no load. Finally, leave your cloud provider's HTTP proxy's timeout set to default. The result is 100% of auth requests timeout once the login queu…

"because Kubernetes"? Is this assuming that you're running your server inside of a Kubernetes instance (and if so, is Kubernetes going to have problems with more than one thread?), or is there some other reason why it comes into this?

Re: Why is everything so scalable?

#300

Earlier quoted context omitted.

Legitimately asking, how? The only bottleneck should be the DB, and if you can saturate a 128-core DB, I want to see your queries and working set size. Not saying it can’t happen, but it’s rare that someone has actually maxed out MySQL or Postgres without there being some serious schema and query flaws, or just poor / absent tuning.

You’re thinking purely in terms of app performance. have you ever seen a terrible db schema? Having to suddenly iterate fast with a brittle codebase that doesnt really allow that ive seen bring teams to their knees for a year+. I’ve seen monoliths because of their sheer size and how much crap and debt is packed into them, build and deploy processes taking several hours if not an entire day for some fix that could be…

> have you ever seen a terrible db schema?

I am a DBRE, so yes, unfortunately most days I see terrible schemata.

> Having to suddenly iterate fast with a brittle codebase that doesnt really allow that ive seen bring teams to their knees for a year+.

IME, the “let’s move fast” mindset causes further problems, because it’s rare that a dev has any inkling about proper data modeling, let alone RDBMS internals. What I usually see are heavily denormalized tables, UUIDs everywhere, and JSON taking the place of good modeling practices. Then they’re surprised when I tell them the issue can’t be fixed with yet another index, or a query rewrite. Turns out when you have the largest instance the cloud provider has, and your working set still doesn’t fit into memory, you’re gonna have a bad time.

Post reply on HN