Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is everything so scalable?

stavros.io

Post reply on HN
Print view View original

301–310 of 383 posts

Re: Why is everything so scalable?

#301

Post by san1t1 »

geoduck14 wrote: post #291
theamk wrote: post #227

Earlier quoted context omitted.

> performance regressions, heat maps, kernel issues etc. > AWS takes that away and makes you focus on the product. ha ha ha no. Have been dealing with kernel issues on my AWS machines for a long time. They lock up under certain kinds of high load. AWS support is useless. Experimenting with kernel version leads to performance regressions. AWS is great if your IT/purchasing department is inefficient. Getting a new AWS…

>AWS is great if your IT/purchasing department is inefficient Fwiw, I think a lot of companies have this problem.

I think the conversation has turned from "Can we spend more?" to "Can you please try and spend less?"

Re: Why is everything so scalable?

#302

Post by toast0 »

gaoshan wrote: post #124
sgarland wrote: post #47

Earlier quoted context omitted.

THANK YOU. People look at me like I’m insane when I tell them that their overly-complicated pipeline could be easily handled by a couple of beefy servers. Or at best, they’ll argue that “this way, they don’t have to manage infrastructure.” Except you do - you absolutely do. It’s just been partially abstracted away, and some parts like OS maintenance are handled (not that that was ever the difficult part of managing s…

Anyone that says, "they don’t have to manage infrastructure" I would invite them to deal with a multi-environment terraform setup and tell me again that about what they don't have to manage.

I've certainly done some things where outsourcing hosting meant I didn't have to manage infrastructure. For services running on vm instances in gcp vs services running on bare metal managed hosts, there's not a whole lot of difference in terms of management IMHO.

But any infrastructure that the product I support use is infrastructure I need to manage; having it outside my control just makes it that much harder to manage. If it's outside my control, the people who control it better do a much better job than I would at managing it, otherwise it's going to be a much bigger pain.

Re: Why is everything so scalable?

#303

Post by dwoldrich »

acron0 wrote: post #4

Ugh, there is just something so satisfying about developer cynicism. It gives me that warm, fuzzy feeling. I basically agree with most of what the author is saying here, and I think that my feeling is that most developers are at least aware that they should resist technical self-pleasure in pursuit of making sure the business/product they're attached to is actually performing. Are there really people out there who st…

I needed to build an internal admin console, not super-scalable, just a handful of business users to start. The SQL database it would access was on-premises, but might move to the cloud in future. Authorized users needed single sign-on to their Azure-based active directory accounts for login. I wanted to do tracing of user requests with OpenTelemetry or something like.

At this point in my career, why wouldn't I reach for microservices to supply the endpoints that my frontend calls out to? Microservices are straightforward to implement with NodeJS (or any other language, for that matter.) I get very straightforward tracing and Azure SSO support in NodeJS. For my admin console, I figured I would need one backend-for-frontend microservice that the frontend would connect to and a domain service for each domain that needed to be represented (with only one domain to start). We picked server technologies and frameworks that could easily port to the cloud.

So two microservices to implement a secure admin console from scratch, is that too many? I guess I lack the imagination to do the project differently. I do enjoy the "API First" approach and the way it lets me engage meaningfully with the business folks to come up with a design before we write any code. I like how it's easy to unit/functional test with microservices, very tidy.

Perhaps what makes a lot/most of microservice development so gross is misguided architectural and deployment goals. Like, having a server/cluster per deployed service is insane. I deploy all of my services monolithically until a service has some unique security or scaling needs that require it to separate from the others.

Similarly, it seems common for microservices teams to keep multiple git repos, one for each service. Why?! Some strange separation-of-concerns/purity ideals. Code reuse, testing, pull requests, and atomic releases suffer needless friction unless everything is kept in a monorepo, as the OP implied.

Also, when teams build microservices in such a way that services must call other services completely misses the point of services - that's just creating a distributed monolith (slow!)

I made a rule on my team that the only service type that can call another service is aggregation services like my backend-for-frontend which could launch downstream calls in parallel and aggregate the results for the caller. This made the architecture very flat with the minimum number of network hops and with as much parallelism as possible so it would stay performant. Domain services owned their data sources, no drama with backend data.

I see a lot of distributed monolith drama and abuse of NoSQL data sources giving microservices a bad reputation.

Re: Why is everything so scalable?

#304

Post by abujazar »

JackSlateur wrote: post #192
BirAdam wrote: post #29

Just to be honest for a bit here... we also should be asking what kind of scale? Quite a while ago, before containers were a thing at all, I did systems for some very large porn companies. They were doing streaming video at scale before most, and the only other people working on video at that scale were Youtube. The general setup for the largest players in that space was haproxy in front of nginx in front of several…

It is an issue to mistake scalability with resiliency Yes, we can run twitter on a single server ( https://thume.ca/2023/01/02/one-machine-twitter/ ) No, we do not want to run twitter on a single server

It's much easier to build a resilient system with a simple architecture. E.g. run the application on a decent VM or even bare metal server and mirror the whole system between a few different data centers.

Re: Why is everything so scalable?

#305

Post by cestith »

jrochkind1 wrote: post #224

> No, no you don’t, you can deploy your modules to their own repos and work on them that way, though then you lose the nice property of being able to atomically deploy changes across your entire codebase when an API changes. That's not all you lose. You also lose being able to have a single git SHA to describe the state of the entire system at any time. And, lose the naturalness of running CI on the entire system at…

I hear you about rigging those repos together artificially.

I used to work somewhere that had several different systems written as monoliths that eventually needed to interact more closely with each other. There was an in-house ticket system (since made a support service wrapped around ZenDesk, but ZD wouldn’t replicate all its functionality). There was an in-house employee management system. There was a first-party CRM. There was a homegrown e-commerce store. They’d built their own licensing servers for their software. Eventually the CRM was managing large customer licenses, the ticket system linked to the store to sell priority support, and the store was selling licenses to smaller customers.

So instead of making these things all support APIs that supported the other applications, people started copying libraries around from service to service. Then to make sure those libraries didn’t fall out of date, the (at the time rsync) deployment process for each of those apps was changed to require a pull from every one of those repos, then a push from that staging server to the production servers. Then security did a PCI-DSS internal audit, and the developers couldn’t just get onto a staging server and make direct changes to production.

So I, as the lead SRE at the time, wrote a builder web app that takes a config file per project. It holds the data on the repos and the default tag to pull from each. The web app allows the developer to update to a different commit or tag for any particular repo involved. Then a single button pulls everything and serially takes production servers out of rotation, updates them, and puts them back into production. It’s something that could have been avoided many different ways including using a monorepo for those systems.

Re: Why is everything so scalable?

#306

Post by thewebguyd »

stavros wrote: post #109
CaptainOfCoit wrote: post #3

I've seen startups killed because of one or two "influential" programmers deciding they need to start architecturing the project for 1000TPS and 10K daily users, as "that's the proper way to build scalable software", while the project itself hasn't even found product-market fit yet and barely has users. Inevitably, the project needs to make a drastic change which now is so painful to do because it no longer fits the…

> 1000TPS and 10K daily users I absolutely agree with your point, but I want to point out, like other commenters here, that the numbers should be much larger. We think that, because 10k daily users is a big deal for a product, they're also a big deal for a small server, but they really aren't. It's fantastic that our servers nowadays can easily handle multiple tens of thousands of daily users on $100/mo.

> We think that, because 10k daily users is a big deal for a product, they're also a big deal for a small server, but they really aren't.

Yeah we seem to forget just how fast computers are now a days. Obviously varies with complexity of the app & what other tech you are using, but for simpler things 10k daily users could be handled by a reasonbly powerful desktop sitting under my desk without breaking a sweat.

Re: Why is everything so scalable?

#307

Post by icedchai »

BobbyTables2 wrote: post #120
sgarland wrote: post #47

Earlier quoted context omitted.

THANK YOU. People look at me like I’m insane when I tell them that their overly-complicated pipeline could be easily handled by a couple of beefy servers. Or at best, they’ll argue that “this way, they don’t have to manage infrastructure.” Except you do - you absolutely do. It’s just been partially abstracted away, and some parts like OS maintenance are handled (not that that was ever the difficult part of managing s…

Have always felt the same. I’ve seen an entire company proudly proclaim a modern multicore Xeon with 32GB RAM can do basic monitoring tasks that should have been possible with little more than an Arduino. Except the 32GB Xeon was far too slow for their implementation...

Let me guess: database tables with no indexes, full scans everywhere?

Re: Why is everything so scalable?

#308

Post by kune »

darkwater wrote: post #150
macNchz wrote: post #130

Earlier quoted context omitted.

Working on various teams operating on infrastructure that ranged from a rack in the back of the office, a few beefy servers in a colo, a fleet of Chef-managed VMs, GKE, ECS, and various PaaSes, what I've liked the most about the cloud and containerized workflows is that they wind up being a forcing function for reproducibility, at least to a degree. While it's absolutely 100% possible to have a "big beefy server arch…

You have to remove admin rights to your admins then, because scrappy enough DevOps/platform engineers/whatever will totally hand-edit your AWS infra or Kubernetes deployments. I suffered that first hand. And it's even worse that in the old days, because at least back in the day it was expected.

In my org nobody has admin rights with the exception of emergencies, but we are ending up with a directory full of Github workflows and nobody knows, which of them are currently supposed to work.

Nothing beats people knowing what they are doing and cleaning up behind them.

Re: Why is everything so scalable?

#309

Post by rvitorper »

koito17 wrote: post #209

In 2015, a single server with 40 cores and 128 GB of RAM was able to handle 2 million WebSocket connections running an Elixir program[0]. One of my hot takes is that a gaming PC has fast enough hardware to serve thousands of clients with a static Rust binary and SQLite. Pair with Litestream, and you have easy-to-test, continuous backups. It's nice being able to test backups by just running `litestream restore` and th…

Can’t argue with that. Machines are pretty capable, and elixir is awesome as well

Re: Why is everything so scalable?

#310

Post by andoando »

potatolicious wrote: post #88
nicoburns wrote: post #18

Earlier quoted context omitted.

I guess the work is deterministic, but it often (unintentionally) makes the systems being developed non-deterministic!

Ah yes. I once worked at a startup that insisted on Mongo despite not having anywhere near the data volume for it to make any sense at all. Like, we're talking 5 orders of magnitude off of what one would reasonably expect to need a Mongo deployment. I was but a baby engineer then, and the leads would not countenance anything as pedestrian as MySQL/Postgres. Anyway, fast forward a bit and we were tasked with building…

Dawg tinder was operating at 20M DAU with all its DB based on Dynamo. Probably still is.

And yeah there was ton of those issues but yolo

Post reply on HN