Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost every infrastructure decision I endorse or regret

cep.dev

Post reply on HN
Print view View original

441–450 of 644 posts

Re: Almost every infrastructure decision I endorse or regret

#441

Post by rnts08 »

matwood wrote: post #342
macNchz wrote: post #195

Earlier quoted context omitted.

In the small SaaS startup case, I’d say the production database is typically the most critical single piece of infra, so self hosting is just not a compelling proposition unless you have a strong technical reason where having super powerful database hardware is important, or a team with multiple people who have sysadmin or DBA experience. I think both of those cases are unusual. I’ve been the guy managing a critical…

I am good at databases (have been a DBA in the past), and 100% agree with this. RDS is easy to standup and get all the things you mentioned, and not have to think about again. If we grow to the point where the overhead is more than a FT DBA, awesome. It means we are successful, and are fortunate to have options.

Unfortunately there are so many people and teams who thinks that simply running their databases on RDS means that they're backed up, highly-available and can be easily load balanced, upgraded, partitioned, migrated and so on which is simply not the case with the basic configuration.

RDS is a great choice, for prototyping and only for production if you know what you're doing when setting it up.

FWIW, this is common in all cloud deployments, people assume that running something "severless" is a magical silver bullet.

Re: Almost every infrastructure decision I endorse or regret

#442

Post by nicbou »

nextos wrote: post #128
nyc_data_geek wrote: post #86

Earlier quoted context omitted.

Some orgs are looking at moving back to on prem because they're figuring this out. For a while it was vogue to go from capex to opex costs, and C suite people were incentivized to do that via comp structures, hence "digital transformation" ie: migration to public cloud infrastructure. Now, those same orgs are realizing that renting computers actually costs more than owning them, when you're utilizing them to a signif…

Same experience here. As a small organization, the quotes we got from cloud providers have always been prohibitively expensive compared to running things locally, even when we accounted for geographical redundancy, generous labor costs, etc. Plus, we get to keep know how and avoid lock-in, which are extremely important things in the long term. Besides, running things locally can be refreshingly simple if you are just…

Is there a bit of risk involved since the know-how has a will of its own and sometimes gets sick?

If I had a small business with very clever people I'd be very afraid of what happens if they're not available for a while.

Re: Almost every infrastructure decision I endorse or regret

#443

Post by annoyingnoob »

ndjshe3838 wrote: post #78

I’m imagining a developer in the 90s/00s reading this list and being baffled by the complexity/terminology

No, not at all. Maybe baffled by the use of expensive cloud services instead of running on your own bare metal where the cost is in datacenter space and bandwidth. The loss of control coupled with the cost is baffling.

Re: Almost every infrastructure decision I endorse or regret

#444

Post by brainzap »

politelemon wrote: post #316

> Ubuntu for dev servers I didn't understand this section. Ubuntu servers as dev environment, what do you mean? As in an environment to deploy things onto, or a way for developers to write code like with VSCode Remote?

My take from this was more: being uniform reduced overhead of maintaining.

Being able to write a bash script that runs on ever machine is nice.

Re: Almost every infrastructure decision I endorse or regret

#445

Post by blantonl »

amluto wrote: post #107
steveBK123 wrote: post #11

Earlier quoted context omitted.

RDS pricing is deranged at the scales I've seen too. $60k/year for something I could run on just a slice of one of my on-prem $20k servers. This is something we would have run 10s of. $600k/year operational against sub-$100k capital cost pays DBAs, backups, etc with money to spare. Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bullet…

I have a small MySQL database that’s rather important, and RDS was a complete failure. It would have cost a negligible amount. But the sheer amount of time I wasted before I gave up was honestly quite surprising. Let’s see: - I wanted one simple extension. I could have compromised on this, but getting it to work on RDS was a nonstarter. - I wanted RDS to _import the data_. Nope, RDS isn’t “SUPER,” so it rejects a bun…

replication to non-AWS systems. "simple" extension problems importing data into RDS because of your custom stuff lurking in a mysqldump

Sounds like you are walking massive edge

Re: Almost every infrastructure decision I endorse or regret

#446

Post by bakchodi »

politelemon wrote: post #335
rswail wrote: post #253

Earlier quoted context omitted.

We are a reasonably large AWS customer and our account manager sends out regular emails with NDA information on what's coming up, we have regular meetings with them about things as wide ranging as database tuning and code development/deployment governance. They often provide that consulting for free, and we know their biases. There's nothing hidden about the fact that they will push us to use AWS services. On the oth…

> There's nothing hidden about the fact that they will push us to use AWS services. Our account team don't even do that. We use a lot of AWS anyway and they know it, so they're happy to help with competitor offerings and integrating with our existing stack. Their main push on us has been to not waste money.

When I was at AWS, I watched SAs get promoted for saving customers money all the time.

AWS wants happy customers to stick around for a long time, not one month of goosed income

Re: Almost every infrastructure decision I endorse or regret

#447

Post by jstummbillig »

davedx wrote: post #267

Utter insanity. So much cost and complexity, and for what? Startups don’t think about costs or runway anymore, all they care about is “modern infrastructure”. The argument for RDS seems to be “we can’t automate backups”. What on earth?

The argument for RDS (and other services along those lines) is "we can't do it as good, for less".

And, when factoring in all costs and considering all things the service takes care of, it seems like a reasonable assumption that in a free market a team that specializes in optimizing this entire operation will sell you a db service at a better net rate than you would be able to achieve on your own.

Which might still turn out to be false, but I don't think it's obvious why.

Re: Almost every infrastructure decision I endorse or regret

#448

Post by from-nibly »

jspdown wrote: post #436
cloogshicer wrote: post #421

Earlier quoted context omitted.

Genuinely curious (I don't have much experiences with DBs), how is schema migration done 'properly' these days?

Incremental forward-only migrations (non-state based). Then, for the How and When, it mostly depends of your constraints and sizes. There's no silver bullet, it's hard, it require constant thinking, it's a slow and often multi step process. I never saw a successful fully automated one-way-of-doing process.

Are you talking about the mechanics? Like more than just run a migration script on boot?

Re: Almost every infrastructure decision I endorse or regret

#449

Post by iandanforth »

For people who enjoyed this post but want to see the other side of the spectrum where self hosted is the norm I'll point to the now classic series of posts on how Stack Overflow runs its infra: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...

If anyone has newer posts like the above, please reply with links as I would love to read them.

Re: Almost every infrastructure decision I endorse or regret

#450

Post by iaresee »

viraptor wrote: post #229
gen220 wrote: post #61

Earlier quoted context omitted.

I’m a big fan of Datadog from multiple angles. Their pricing setup is evil. Breaking out by SKUs and having 10+ SKUs is fine, trialing services with “spot” prices before committing to reserved capacity is also fine. But (for some SKUs, at least) they make it really difficult to be confident that the reserved capacity you’re purchasing will cover your spot use cases. Then, they make you contact a sales rep to lower yo…

Another annoying thing is that the billing dashboards do not map clearly to what's on the pricing pages / in the contract. Good luck figuring out the extras for RUM when you have multiple orgs. Then they have things that I wanted to try for a long time, but... support doesn't care? Repeated "would you like to use this? / very likely, can we try it out? / (silence)". I love their product, but they are so annoying to d…

> Another annoying thing is that the billing dashboards do not map clearly to what's on the pricing pages / in the contract. Good luck figuring out the extras for RUM when you have multiple orgs.

I, quite literally, was griping to my Datadog CSM about this exact thing last week. They'll email me and be, "Oh, you know you're logging volume this month put you into on-demand indexing rates, right?" and my answer is always, "No, because your monitoring platform makes it nearly impossible for me to monitor it correctly."

You can't reference your contracted volume rates when building monitors out and the units for the metrics you need to watch don't match the units you contract with them on the SKU.

Maddening.

Post reply on HN