Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost every infrastructure decision I endorse or regret

cep.dev

Post reply on HN
Print view View original

31–40 of 644 posts

Re: Almost every infrastructure decision I endorse or regret

#31

Post by macNchz »

steveBK123 wrote: post #11
electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

RDS pricing is deranged at the scales I've seen too. $60k/year for something I could run on just a slice of one of my on-prem $20k servers. This is something we would have run 10s of. $600k/year operational against sub-$100k capital cost pays DBAs, backups, etc with money to spare. Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bullet…

> Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bulletproof.. it makes sense? But how many cases are there of this?

Most software startups these days? The blog post is about work done at a startup after all. By the time your db is big enough to cost an unreasonable amount on RDS, you’re likely a big enough team to have options. If you’re a small startup, saving a couple hundred bucks a month by self managing your database is rarely a good choice. There’re more valuable things to work on.

Re: Almost every infrastructure decision I endorse or regret

#32

Post by z3ugma »

breckenedge wrote: post #23

> There are no great FaaS options for running GPU workloads, which is why we could never go fully FaaS. I keep wondering when this is going to show up. We have a lot of service providers, but even more frameworks, and every vendor seems to have their own bespoke API.

Check out beam.cloud. They’re impressing me with calling GPU runtimes as a FaaS

Re: Almost every infrastructure decision I endorse or regret

#33

Post by sgarland »

Scubabear68 wrote: post #27
sgarland wrote: post #15

Earlier quoted context omitted.

The downside is then you have many, many DBs to fight with, to monitor, to tune, etc. This is rarely a problem when things are small, but as they grow, the bad schema decisions made by empowering DBA-less teams to run their own infra become glaringly obvious.

Not a downside to me. Each team maintains their own DB and pays for their own choices. In the kitchen sink model all teams are tied together for performance and scalability, and some bad apple applications can ruin the party for everyone. Seen this countless times doing due diligence on startups. The universal kitchen sink DB is almost always one of the major tech debt items.

I’m a DBRE, which means it’s somehow always my fault until proven otherwise. And even then, it’s usually on me to work around the insane schema dreamt up by the devs.

Multi-tenant DBs can work fine as long as every app has its own users, everyone goes through a connection pooler / load balancer, and every user has rate limits. You want to write shitty queries that time out? Not my problem. Your GraphQL BFF bullshit is trying to make 10,000 QPS? Nope, sorry, try again later.

EDIT: I say “not my problem,” but as mentioned, it inevitably becomes my problem. Because “just unblock them so the site is functional” is far more attractive to the C-Suite than “slow down velocity to ensure the dev teams are doing things right.”

Re: Almost every infrastructure decision I endorse or regret

#34

Post by vrosas »

sgarland wrote: post #15
Scubabear68 wrote: post #3

The kitchen sink database used by everybody is such a common problem, yet it is repeated over and over again. If you grow it becomes significant tech debt and a performance bottleneck. Fortunately, with managed DBs like RDS it is really easy to run individual DB clusters per major app.

The downside is then you have many, many DBs to fight with, to monitor, to tune, etc. This is rarely a problem when things are small, but as they grow, the bad schema decisions made by empowering DBA-less teams to run their own infra become glaringly obvious.

Bad schema decisions are made regardless of whether you’re one database or 50. At least with many databases the problems are localized.

Re: Almost every infrastructure decision I endorse or regret

#35

Post by renewiltord »

sgarland wrote: post #30
renewiltord wrote: post #9

Earlier quoted context omitted.

You don't get the higher end machines on AWS unless you're a big guy. We have Epyc 9684X on-prem. Cannot match that at the price on AWS. That's just about making the choices. Most companies are not DB-primary.

I think most people who’ve never experienced native NVMe for a DB are also unaware of just how blindingly fast it is. Even io2 Block Express isn’t the same.

Yes. We have it 4x striped on those same machines. Burns like lightning.

Re: Almost every infrastructure decision I endorse or regret

#36

Post by [deleted] »

macNchz wrote: post #31
steveBK123 wrote: post #11

Earlier quoted context omitted.

RDS pricing is deranged at the scales I've seen too. $60k/year for something I could run on just a slice of one of my on-prem $20k servers. This is something we would have run 10s of. $600k/year operational against sub-$100k capital cost pays DBAs, backups, etc with money to spare. Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bullet…

> Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bulletproof.. it makes sense? But how many cases are there of this? Most software startups these days? The blog post is about work done at a startup after all. By the time your db is big enough to cost an unreasonable amount on RDS, you’re likely a big enough team to have options. If yo…

[deleted]

Re: Almost every infrastructure decision I endorse or regret

#37

Post by sgarland »

vrosas wrote: post #34
sgarland wrote: post #15

Earlier quoted context omitted.

The downside is then you have many, many DBs to fight with, to monitor, to tune, etc. This is rarely a problem when things are small, but as they grow, the bad schema decisions made by empowering DBA-less teams to run their own infra become glaringly obvious.

Bad schema decisions are made regardless of whether you’re one database or 50. At least with many databases the problems are localized.

But then the DB Team – if you have one – is responsible for 50 databases, each full of their own unique problems.

This will undoubtedly go over poorly, but honestly I think every data decision should be gated through the DB Team (again, if you have them). Your proposed schema isn’t normalized? Straight to jail. You don’t want to learn SQL? Also straight to jail. You want to use a UUIDv4 as a primary key? Believe it or not, jail.

The most performant and referentially sound app in the world, because of jail.

Re: Almost every infrastructure decision I endorse or regret

#38

Post by sgarland »

renewiltord wrote: post #35
sgarland wrote: post #30

Earlier quoted context omitted.

I think most people who’ve never experienced native NVMe for a DB are also unaware of just how blindingly fast it is. Even io2 Block Express isn’t the same.

Yes. We have it 4x striped on those same machines. Burns like lightning.

The only problem is it hides all of the horrible queries. Ah well, can’t have it all.
Post reply on HN