Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is everything so scalable?

stavros.io

Post reply on HN
Print view View original

351–360 of 383 posts

Re: Why is everything so scalable?

#351

Post by DanielHB »

vidarh wrote: post #248
mystifyingpoi wrote: post #215

Earlier quoted context omitted.

The parent compares RDS to baremetal, which I think isn't a fair comparison at all. Especially since we don't know the specs of either of these. I found RDS to be rock solid too, although performance issues are often resolved by developers by submitting a PR that bumps the instance size x2, because "why not". On baremetal it's often impossible to upgrade CPU just like that, so people have to fix performance issues el…

RDS works great, but it's far easier to scale a bare metal setup to an extent that makes RDS look like an expensive toy because you have far more hardware options RDS is a good option if you want convenience and simplicity, though.

Managing database backups myself is something that gives me nightmares. I would refuse to use bare-metal dbs unless I have a dedicated team just to manage the database (or data that is okay to lose, like caching layers).

Re: Why is everything so scalable?

#352

Post by DanielHB »

crdrost wrote: post #234
icedchai wrote: post #195

Earlier quoted context omitted.

I agree that building your backend on Lambda is terrible for many reasons: slow starts, request / response size restrictions, limitations in "layer" sizes, etc. RDS, however, I have found to be rock solid. What have you run into?

I don't know too much about the performance side of RDS, but the backup model is absolutely a headache. It's at the point where I'd rather pg_dump into gz and upload to s3.

I never had problems with RDS backup, but I never used it a ton or had really large DBs in it. What problems did you ran into?

Re: Why is everything so scalable?

#353

Post by DanielHB »

geoduck14 wrote: post #291
theamk wrote: post #227

Earlier quoted context omitted.

> performance regressions, heat maps, kernel issues etc. > AWS takes that away and makes you focus on the product. ha ha ha no. Have been dealing with kernel issues on my AWS machines for a long time. They lock up under certain kinds of high load. AWS support is useless. Experimenting with kernel version leads to performance regressions. AWS is great if your IT/purchasing department is inefficient. Getting a new AWS…

>AWS is great if your IT/purchasing department is inefficient Fwiw, I think a lot of companies have this problem.

My wife is literally consulting for a big bank where it takes them 6+ months to get an on-premise VM setup and configured.

Re: Why is everything so scalable?

#354

Post by vidarh »

DanielHB wrote: post #351
vidarh wrote: post #248

Earlier quoted context omitted.

RDS works great, but it's far easier to scale a bare metal setup to an extent that makes RDS look like an expensive toy because you have far more hardware options RDS is a good option if you want convenience and simplicity, though.

Managing database backups myself is something that gives me nightmares. I would refuse to use bare-metal dbs unless I have a dedicated team just to manage the database (or data that is okay to lose, like caching layers).

Managing database backups is fairly straightforward. Postgres + a base backup + long term wal archiving in a blob store is very easy to set up and monitor. It could be easier, and if you don't want to manage that using RDS is certainly a valid choice, but it's a tradeoff - I often have customers that help addressing performance issues with RDS they simply wouldn't have if they sized a bare metal setup with enough RAM and NVMe and configured it even halfway decently instead, and the end result is often that they end up paying more for devops help to figure out performance bottlenecks than they'd spend putting the same devops consultant on retainer ensuring they have a solid backup setup.

Re: Why is everything so scalable?

#355

Post by whstl »

JohnMakin wrote: post #336
whstl wrote: post #265

Earlier quoted context omitted.

That was the reality of the fintech I worked at. The schema wasn't really a problem, but the sheer amount of queries per request. Often a user opening a page or clicking a button would cause 100-200 database queries, including updates. This would prevent strategies such as "just replicating the data somewhere". It was so badly architected that every morning the app would stop responding due to users doing their morni…

sounds so similar to a fintech situation I’ve been in once I swear I was gonna say we worked at the same place, but the size of company is wrong. I’ve now seen pretty similar things since, enough to say it’s probably everywhere

I know some people working at very popular EU fintech-banks and the description I get is that it’s just a shitshow engineering wise.

Good engineering is unfortunately at odds with modern development practices.

Re: Why is everything so scalable?

#357

Post by darkwater »

Aeolun wrote: post #286
darkwater wrote: post #150

Earlier quoted context omitted.

You have to remove admin rights to your admins then, because scrappy enough DevOps/platform engineers/whatever will totally hand-edit your AWS infra or Kubernetes deployments. I suffered that first hand. And it's even worse that in the old days, because at least back in the day it was expected.

But then your next deployment goes, and it all rolls back, right? And then it their fault, right? I might have mild trauma from people complaining their artisanal changes to our environment weren’t preserved.

Weeeeell, if you use Helm the manual change might be preserved, which makes investigations even more... interesting.

Re: Why is everything so scalable?

#358

Post by dagss »

mmcnl wrote: post #240

Scalability is not purely technical. It's also organizational. For all its drawbacks, the microservices architecture is easier to scale from an organizational perspective.

My experience is it adds a multiplier 10x to the work that needs to be done, which is then a good fit for the 9x extra developers upper management decided to hire.

Sure, it keeps everyone busy with work they feel is meaningful -- but it really is work that was not really needed in the first place with a monolith.

Much simpler to try to stay fewer developers as long as possible.

Also in my experience it is impossible to get people along with organizing the company along software architecture division lines, thus creating a lot of political problems.

Re: Why is everything so scalable?

#359

Post by john_the_writer »

The only bit that I didn't agree with was the downsides. You can scale individual modules. You can set your load balancer to make all calls to specific endpoints go to specific servers. These can scale independently.

If you have background workers (ActiveJobs/Oban for example), these can be on different queues, that you can scale. It's actually really easy to build out a mono-repo system allow for scale.

If you organise your workers into folders based on their purpose (reporting, exporting, ... ), and you're careful about feature flags, you can drastically reduce git-conflicts and CICD issues.

Re: Why is everything so scalable?

#360

Post by brokegrammer »

I agree with this post, but how do we convince decision makers to use simpler solutions? Here's what happened to me not long ago:

A person contacted me to build a tool that will have a single user. After some questioning, the person said they would likely have up to 3 users for this tool at some point, and no more.

So I suggested we use Python and a SQlite database. When I reached out after not having heard from them for 2 weeks, they said that they went with a more experienced programmer because the latter suggested MongoDB, a separate ReactJS front-end, and micro services with Golang. All deployed to AWS, of course.

In the end, I lost the gig because I wanted to save the client time and money.

Post reply on HN