Almost every infrastructure decision I endorse or regret • Hacker News Board

Re: Almost every infrastructure decision I endorse or regret

#311

Post by fullstackchris » Sat, Feb 10, 2024, 9:56 AM UTC

davedx wrote: post #303

isbvhodnvemrwvn wrote: post #292

Earlier quoted context omitted.

Is spending time to make it reliable worth it vs working on your actual product? Databases are THE most critical things your company has.

All that infra doesn’t integrate itself. Everywhere I’ve worked that had this kind of stack employed at least one if not a team of DevOps people to maintain it all, full time, the year round. Automating a database backup and testing it works takes half a day unless you’re doing something weird

A startup sized company using this many tools? They're for sure doing something weird (and that's not a compliment :) )

Totally on your side with this one - but alas, people associate value with complexity.

Re: Almost every infrastructure decision I endorse or regret

#312

Post by kosolam » Sat, Feb 10, 2024, 10:04 AM UTC

What is the cost? With 1/10th of the sum one capable engineer can setup a way better infra on premise. The days of free money is over, guys. Wake up!

Re: Almost every infrastructure decision I endorse or regret

#313

Post by ffsm8 » Sat, Feb 10, 2024, 10:07 AM UTC

davedx wrote: post #303

isbvhodnvemrwvn wrote: post #292

Earlier quoted context omitted.

Is spending time to make it reliable worth it vs working on your actual product? Databases are THE most critical things your company has.

All that infra doesn’t integrate itself. Everywhere I’ve worked that had this kind of stack employed at least one if not a team of DevOps people to maintain it all, full time, the year round. Automating a database backup and testing it works takes half a day unless you’re doing something weird

> Automating a database backup and testing it works takes half a day unless you’re doing something weird

True story bro

I'm sure that's possible if you're storing the backup on the same server you're restoring on and everything is on top of the line nvme storage. Otherwise your backup just started to run and will need another few days to finish. And that's only if you're running single master.

You're massively underestimating the challenge to get that kind of automation done in a stable manner - and the maintenance required to keep it working over the years.

Re: Almost every infrastructure decision I endorse or regret

#314

Post by graemep » Sat, Feb 10, 2024, 10:12 AM UTC

theptip wrote: post #162

tw04 wrote: post #130

Earlier quoted context omitted.

>By the time your db is big enough to cost an unreasonable amount on RDS, you’re likely a big enough team to have options. By the time your db is big enough to cost an unreasonable amount on RDS, you've likely got so much momentum that getting off is nearly impossible as you bleed cash. You can buy a used server and find colocation space and still be pennies on the dollar for even the smallest database. If you're doi…

That’s just another way of saying the opportunity cost isn’t worth paying to do the migration. Optionality and flexibility are extremely valuable, and that is why cloud compute continues to be popular, especially for rapidly/burstily growing businesses like startups.

People do not really understand the value of the former. Even dealing with financial options (buy/sell and underlying) which are a pure form of it, people either do not understand the value, or do so in a very abstract way they do not intuit.

Re: Almost every infrastructure decision I endorse or regret

#315

Post by silisili » Sat, Feb 10, 2024, 10:14 AM UTC

electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

Even for small workloads it's a difficult choice. I ran a small but vital db, and RDS was costing us like 60 bucks a month per env. That's 240/month/app.

DynamoDB as a replacement, pay per request, was essentially free.

I found Dynamo foreign and rather ugly to code for initially, but am happy with the performance and especially price at the end.

Re: Almost every infrastructure decision I endorse or regret

#316

Post by politelemon » Sat, Feb 10, 2024, 10:15 AM UTC

> Ubuntu for dev servers

I didn't understand this section. Ubuntu servers as dev environment, what do you mean? As in an environment to deploy things onto, or a way for developers to write code like with VSCode Remote?

Re: Almost every infrastructure decision I endorse or regret

#317

Post by hhsectech » Sat, Feb 10, 2024, 10:16 AM UTC

jumploops wrote: post #280

nyc_data_geek wrote: post #86

Earlier quoted context omitted.

Some orgs are looking at moving back to on prem because they're figuring this out. For a while it was vogue to go from capex to opex costs, and C suite people were incentivized to do that via comp structures, hence "digital transformation" ie: migration to public cloud infrastructure. Now, those same orgs are realizing that renting computers actually costs more than owning them, when you're utilizing them to a signif…

Funny story time. I was once part of an acquisition from a much larger corporate entity. The new parent company was in the middle of a huge cloud migration, and as part of our integration into their org, we were required to migrate our services to the cloud. Our calculations said it would cost 3x as much to run our infra on the cloud. We pushed back, and were greenlit on creating a hybrid architecture that allowed us…

There are two possible scenarios here. Firstly, they can't find the talent to support what you implemented...or more likely, your docs suck!

I've made a career out of inheriting other peoples whacky setups and supporting them (as well as fixing them) and almost always its documentation that has prevented the client getting anywhere.

I personally dont care if the docs are crap because usually the first thing I do is update / actually write the docs to make them usable.

For a lot of techs though crap documentation is a deal breaker.

Crap docs aren't always the fault of the guys implementing though, sometimes there are time constraints that prevent proper docs being written. Quite frequently though its outsourced development agencies that refuse to write it because its "out of scope" and a "billable extra". Which I think is an egregious stance...doxs Should be part and parcel of the project. Mandatory.

Re: Almost every infrastructure decision I endorse or regret

#318

Post by kunley » Sat, Feb 10, 2024, 10:17 AM UTC

steveBK123 wrote: post #11

electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

RDS pricing is deranged at the scales I've seen too. $60k/year for something I could run on just a slice of one of my on-prem $20k servers. This is something we would have run 10s of. $600k/year operational against sub-$100k capital cost pays DBAs, backups, etc with money to spare. Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bullet…

RDS is not so bulletproof as advertised, and the support is first arrogant then (maybe) helpful.

People pay for RDS because they want to believe in a fairy tale that it will keep potential problems away and that it worked well for other customers. But those mythical other customers also paid based on such belief. Plus, no one wants to admit that they pay money in such irrational way. It's a bubble

Re: Almost every infrastructure decision I endorse or regret

#319

Post by hhsectech » Sat, Feb 10, 2024, 10:17 AM UTC

smitty1e wrote: post #307

jumploops wrote: post #280

Earlier quoted context omitted.

Funny story time. I was once part of an acquisition from a much larger corporate entity. The new parent company was in the middle of a huge cloud migration, and as part of our integration into their org, we were required to migrate our services to the cloud. Our calculations said it would cost 3x as much to run our infra on the cloud. We pushed back, and were greenlit on creating a hybrid architecture that allowed us…

My funny story is built on the idea that AWS is Hotel California for your data. A customer had an interest in merging the data from an older account into a new one, just to simplify matters. Enterprise data. Going back years. Not even leaving the region. The AWS rep in the meeting kinda pauses, says: "We'll get back to you on the cost to do that." The sticker shock was enough that the customer simply inherited the ol…

Eh? I've never had a problem moving data out of AWS.

Have people lost the ability to write export and backup scripts?

Re: Almost every infrastructure decision I endorse or regret

#320

Post by politelemon » Sat, Feb 10, 2024, 10:19 AM UTC

> homebrew for Linux

No, just no. I see this cropping up now and then. Homebrew is unsafe for Linux, and is only recommended by Mac users that don't want to bother to learn about existing package management.