Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost every infrastructure decision I endorse or regret

cep.dev

Post reply on HN
Print view View original

121–130 of 644 posts

Re: Almost every infrastructure decision I endorse or regret

#121

Post by dalyons »

eduction wrote: post #76
Scubabear68 wrote: post #3

The kitchen sink database used by everybody is such a common problem, yet it is repeated over and over again. If you grow it becomes significant tech debt and a performance bottleneck. Fortunately, with managed DBs like RDS it is really easy to run individual DB clusters per major app.

Management problem masquerading as a tech problem. Being shared between applications is literally what databases were invented to do. That’s why you learn a special dsl to query and update them instead of just doing it in the same language as your application. The problem is that data is a shared resource. The database is where multiple groups in an organization come together to get something they all need. So it nee…

Why would you make it a shared resource if you don’t have to?

Decades of experience have shown us the massive costs of doing so - the crippled velocity and soul crushing agony of dba change control teams, the overhead salary of database priests, the arcane performance nightmares, the nuclear blast radius, the fundamental organizational counter-incentives of a shared resource .

Why on earth would we choose to pay those terrible prices in this day and age, when infrastructure is code, managed databases are everywhere and every team can have their own thing. You didn’t have a choice previously, now you do.

Re: Almost every infrastructure decision I endorse or regret

#122

Post by cyounkins »

I've climbed the mountain of learning the basics of kubernetes / EKS, and I'm thinking we're going to switch to ECS. Kubernetes is way too complicated for our needs. It wants to be in control and is hard to direct with eg CloudFormation. Load balancers are provisioned from the add-on, making it hard to reference them outside kubernetes. Logging on EKS Fargate to Cloudwatch appears broken, despite following the docs. CPU/Memory metrics don't work like they do on EKS EC2, it appears to require ADOT.

I recreated the environment in ECS in 1/10th the time and everything just worked.

Re: Almost every infrastructure decision I endorse or regret

#123

Post by dalyons »

sgarland wrote: post #33
Scubabear68 wrote: post #27

Earlier quoted context omitted.

Not a downside to me. Each team maintains their own DB and pays for their own choices. In the kitchen sink model all teams are tied together for performance and scalability, and some bad apple applications can ruin the party for everyone. Seen this countless times doing due diligence on startups. The universal kitchen sink DB is almost always one of the major tech debt items.

I’m a DBRE, which means it’s somehow always my fault until proven otherwise. And even then, it’s usually on me to work around the insane schema dreamt up by the devs. Multi-tenant DBs can work fine as long as every app has its own users, everyone goes through a connection pooler / load balancer, and every user has rate limits. You want to write shitty queries that time out? Not my problem. Your GraphQL BFF bullshit i…

Or, you just avoid doing multi tenet from the start and none of those become your problem to unblock. What’s the downside?

Re: Almost every infrastructure decision I endorse or regret

#124

Post by dalyons »

marcosdumay wrote: post #109
inquist wrote: post #88

Earlier quoted context omitted.

What’s wrong with uuidv4 as PK?

Serial integers always work better than any uuid as PKs, but the thing with uuid4 is that it disrupts any kind of index or physical ordering you decide to put on your data. Uuids are really for external communication, not in-system organization.

FWIW this isn’t true anymore with newer uuid schemes like v7 that are roughly time sortable.

Re: Almost every infrastructure decision I endorse or regret

#125

Post by charred_patina »

LispSporks22 wrote: post #74

I feel like this is overkill for a startup. Why not dump your application server and dependencies into rented data center (or EC2 if you must) and setup a coarse DR? Maybe start with a monolith in PHP or Rails. None of that word salad sounds like startup to me, but then again everyone loves to refer to themselves as a startup (must be a recruiting tool?), so perhaps muh dude is spot on.

I don't want to be negative, but this post reads like a list of things that I want to avoid in my career. I did a brief stint in cloud stuff at a FAANG and I don't care to go back to it.

Right now I'm engineer No. 1 at a current startup just doing DDD with a Django monolith. I'm still pretty Jr. and I'm wondering if there's a way to scale without needing to get into all of the things the author of this article mentions. Is it possible to get to a $100M valuation without needing all of this extra stuff? I realize it varies from business to business, but if anyone has examples of successes where people just used simple architecture's I'd appreciate it.

Re: Almost every infrastructure decision I endorse or regret

#126

Post by jacurtis »

infecto wrote: post #65
electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

The problem you have here is by the time you reach the size of this DB, you are on a special discount rate within AWS.

Discount rates are actually much better too on the bigger instances. Therefore the "sticker price" that people compare on the public site is no where close to a fair comparison.

We technically aren't supposed to talk about pricing publically, but I'm just going to say that we run a few 8XL and 12Xl RDS instances and we pay ~40% off the sticker price.

If you switch to Aurora engine the pricing is absurdly complex (its basically impossible to determine without a simulation calculator) but AWS is even more aggressive with discounting on Aurora, not to mention there are some legit amazing feature benefits by switching.

I'm still in agreeance that you could do it cheaper yourself at a Data Center. But there are some serious tradeoffs made by doing it that way. One is complexity and it certainly requires several new hiring decisions. Those have their own tangible costs, but there are a huge amount of intangible costs as well like pure inconvenience, more people management, more hiring, split expertise, complexity to network systems, reduce elasticity of decisions, longer commitments, etc.. It's harder to put a price on that.

When you account for the discounts at this scale, I think the cost gap between the two solutions is much smaller and these inconveniences and complexities by rolling it yourself are sometimes worth bridging that smaller gap in cost in order to gain those efficiencies.

Re: Almost every infrastructure decision I endorse or regret

#127

Post by amluto »

> Zero Trust VPN

VPNs can be wonderful, and you can use use Tailscale or AWS VPN or OpenVPN or IPSEC and you can authenticate using Okta or GSuite or Auth0 or Keycloak or Authelia.

But since when is this Zero Trust? It takes a somewhat unusual firewall scheme to make a VPN do anything that I would seriously construe as Zero Trust, and getting authz on top of that is a real PITA.

Re: Almost every infrastructure decision I endorse or regret

#128

Post by nextos »

nyc_data_geek wrote: post #86
electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

Some orgs are looking at moving back to on prem because they're figuring this out. For a while it was vogue to go from capex to opex costs, and C suite people were incentivized to do that via comp structures, hence "digital transformation" ie: migration to public cloud infrastructure. Now, those same orgs are realizing that renting computers actually costs more than owning them, when you're utilizing them to a signif…

Same experience here. As a small organization, the quotes we got from cloud providers have always been prohibitively expensive compared to running things locally, even when we accounted for geographical redundancy, generous labor costs, etc. Plus, we get to keep know how and avoid lock-in, which are extremely important things in the long term.

Besides, running things locally can be refreshingly simple if you are just starting something and you don't need tons of extra stuff, which becomes accidental complexity between you, the problem, and a solution. This old post described that point quite well by comparing Unix to Taco Bell: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra.... See HN discussion: https://news.ycombinator.com/item?id=10829512.

I am sure for some use-cases cloud services might be worth it, especially if you are a large organization and you get huge discounts. But I see lots of business types blindly advocating for clouds, without understanding costs and technical tradeoffs. Fortunately, the trend seems to be plateauing. I see an increasing demand for people with HPC, DB administration, and sysadmin skills.

Re: Almost every infrastructure decision I endorse or regret

#129

Post by callalex »

steveBK123 wrote: post #66
viraptor wrote: post #50

Earlier quoted context omitted.

Lots of cases. It doesn't even have to be a tiny database. Within Also, Aurora gives you the block level cluster that you can't deploy on your own - it's way easier to work with than the usual replication.

Once you commit to more deeply Amazon flavored parts of AWS like Aurora, aren't you now fairly committed to hoping your scale never exceeds the cost-benefit tradeoff?

If you’re paying list price at scale you are doing it very wrong.

Re: Almost every infrastructure decision I endorse or regret

#130

Post by tw04 »

macNchz wrote: post #31
steveBK123 wrote: post #11

Earlier quoted context omitted.

RDS pricing is deranged at the scales I've seen too. $60k/year for something I could run on just a slice of one of my on-prem $20k servers. This is something we would have run 10s of. $600k/year operational against sub-$100k capital cost pays DBAs, backups, etc with money to spare. Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bullet…

> Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bulletproof.. it makes sense? But how many cases are there of this? Most software startups these days? The blog post is about work done at a startup after all. By the time your db is big enough to cost an unreasonable amount on RDS, you’re likely a big enough team to have options. If yo…

>By the time your db is big enough to cost an unreasonable amount on RDS, you’re likely a big enough team to have options.

By the time your db is big enough to cost an unreasonable amount on RDS, you've likely got so much momentum that getting off is nearly impossible as you bleed cash.

You can buy a used server and find colocation space and still be pennies on the dollar for even the smallest database. If you're doing more than prototyping, you're probably wasting money.

Post reply on HN