Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost every infrastructure decision I endorse or regret

cep.dev

Post reply on HN
Print view View original

421–430 of 644 posts

Re: Almost every infrastructure decision I endorse or regret

#421

Post by cloogshicer »

f549abd0 wrote: post #353
f549abd0 wrote: post #341

Disagree on the point and reasoning about the single database. Sounds like they experienced badly managed and badly constrained database. The described fks and relations: that's what the key constraints and other guard rails and cascades are for - so that you are able to manage a schema. That's exactly how you do it: add in new tables that reference old data. I think the regret is actually not managing the database,…

Reading further > Endorse-ish: Schema migration by Diff Well that explains it... What a terrible approach to migrations for data integrity.

Genuinely curious (I don't have much experiences with DBs), how is schema migration done 'properly' these days?

Re: Almost every infrastructure decision I endorse or regret

#422

Post by hardolaf »

tpetry wrote: post #265
chii wrote: post #249

Earlier quoted context omitted.

i would imagine that cloud infrastructure has the ability for fast scale up, unlike self-owned infrastructure. For example, how long does it take to rent another rack that you didnt plan for? And not to mention that the cost of cloud management platforms that you have to deploy to manage these owned assets is not free. I mean, how come even large consumers of electricity does not buy and own their own infrastructure…

Ordering that amount of amount of servers takes about one hour with hetzner. If you truly want a complete rack on your own maybe a few days as they have to do it manually. Most companies don‘t need to scale up full racks in seconds. Heck, even weeks would be ok for most of them to get new hardware delivered. The cloud planted the lie into everyone‘s head that most companies dont have predictable and stable load.

Most businesses could probably know server needs 6-12 months out. There's a small number of businesses in the world that actually need dynamic scaling.

Re: Almost every infrastructure decision I endorse or regret

#423

Post by maxrecursion »

hhsectech wrote: post #317
jumploops wrote: post #280

Earlier quoted context omitted.

Funny story time. I was once part of an acquisition from a much larger corporate entity. The new parent company was in the middle of a huge cloud migration, and as part of our integration into their org, we were required to migrate our services to the cloud. Our calculations said it would cost 3x as much to run our infra on the cloud. We pushed back, and were greenlit on creating a hybrid architecture that allowed us…

There are two possible scenarios here. Firstly, they can't find the talent to support what you implemented...or more likely, your docs suck! I've made a career out of inheriting other peoples whacky setups and supporting them (as well as fixing them) and almost always its documentation that has prevented the client getting anywhere. I personally dont care if the docs are crap because usually the first thing I do is u…

"Crap docs aren't always the fault of the guys implementing though, sometimes there are time constraints that prevent proper docs being written."

I can always guarantee a stream of consciousness one note that should have most of the important data, and a few docs about the most important parts. It's up to management if they want me to spend time turning that one note into actual robust documentation that is easily read.

Re: Almost every infrastructure decision I endorse or regret

#424

Post by maccard »

sroussey wrote: post #157
maccard wrote: post #103

Earlier quoted context omitted.

Because I can go from main.go to a load balanced, autoscaling app with rolling deploys, segeregated environments, logging & monitoring in about 30 minutes, and never need to touch _any_ of that again. Plus, if I leave, the guy who comes after me can look at a helm chart, terraform module + pipeline.yml and figure out how it works. Meanwhile, our janq shell script based task scheduler craps out on something new every…

I spent zero hours on a MySQL server on bare hardware for seven years. Admittedly, I was afraid of ever restarting as I wasn’t sure it would reboot. But still…

You still need to get mysql installed and configured though. On AWS, it's 30 lines of terraform for RDS on an internal subnet with a security group only allowing access from your cluster.

For that, you get automated backups, very simple read proxies, managed updates of you ever need them. You can vertically scale down, or uo to the point of "it's cheaper to hire a DBA to fix this".

Re: Almost every infrastructure decision I endorse or regret

#425

Post by maccard »

Sankozi wrote: post #407
maccard wrote: post #96

Earlier quoted context omitted.

> Not a downside to me. Each team maintains their own DB and pays for their own choices. This is how you end up with the infamous "jira and confluence have two different markdown flavors" issue.

I don't think Jira and Confluence different markdown setup is due to them not sharing their databases. It is just poor product management from Attlasian.

My point is that forcing these arbitrary decisions is poor product management.

Re: Almost every infrastructure decision I endorse or regret

#426

Post by sseagull »

fswd wrote: post #221

stuff like this makes me want to experiment with going back to just one huge $100k server and running it all on one box in a server rack.

I am doing that. I am part of a research group, and don’t have the $$ or ability to pay so much for all these services.

So we got a $90k server with 184TB of raw storage (SAS SSD), 64 cores, and 1TB of memory. Put it on a 10GB line at our university and it is rock solid. We probably have less downtime than Github, even with reboots every few months.

Have some large (multi-TB) databases on it and web APIs for accessing the data. Would be hugely expensive in the cloud with, especially with egress costs.

You have to be comfortable sys-admining though. Fortunately I am.

Re: Almost every infrastructure decision I endorse or regret

#427

Post by mkesper »

leetharris wrote: post #101
calvinmorrison wrote: post #42

Earlier quoted context omitted.

It's because I hate databases and programming separately. I would rather slow code then have to dig into some database procdure. Its just another level of separation thats too mentally hard to manage. Its like... my queries go into a VM and now I have to worry about how the VM is performing. I wish and maybe there is a programming language with first class database support. I mean really first class not just let me r…

The closest thing to what you're describing is Prisma in Node. It generates a Typescript file from your schema so you get code completion on your data. And it exists somewhere between a query builder and a traditional ORM. I have worked in many languages with many ORMs and this has been my personal favorite.

But the migration stuff is a horrible joke. No way to just rollback a broken migration. https://www.prisma.io/docs/orm/prisma-migrate/workflows/gene...

Re: Almost every infrastructure decision I endorse or regret

#428

Post by gregw2 »

marcosdumay wrote: post #105
el_benhameen wrote: post #59

Earlier quoted context omitted.

Lots of interesting comments on this one. Anyone have any good resources for learning how not to fuck up schema/db design for those of us who will probably never have a DBA on the team?

> not to fuck up schema/db design The neat thing is, you don't. Nobody ever avoids fucking up db design. The best you can do is decide what is really important to get right, and not fuck that part up.

Wow, what an astute comment! Thank you!

P.S. to the original person concerned about this though… for your own sake and your successors, please keep trying.

Re: Almost every infrastructure decision I endorse or regret

#429

Post by hardolaf »

emseetech wrote: post #413
electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

From what I’ve read, a common model for mmorpg companies is to use on-prem or colocated as their primary and then provision a cloud service for backup or overage. Seems like a solid cost effective approach for when a company reaches a certain scale.

Lots of companies, like Grinding Gear Games and Square Enix, just rent whole servers for a tiny fraction of the price compared to what the price gouging cloud providers would charge for the same resources. They get the best of both worlds. They can scale up their infrastructure in hours or even minutes and they can move to any other commodity hardware in any other datacenter at the drop of a hat if they get screwed on pricing. Migrating from one server provider (such as IBM) to another (such as Hetzner) can take an experienced team 1-2 weeks at most. Given that pricing updates are usually given 1-3 quarters ahead at a minimum, they have massive leverage over their providers because they an so easily switch. Meanwhile, if AWS decides to jack up their prices, well you're pretty much screwed in the short-term if you designed around their cloud services.

Re: Almost every infrastructure decision I endorse or regret

#430

Post by geraldhh »

habinero wrote: post #355
LispSporks22 wrote: post #89

Earlier quoted context omitted.

I agree. I’m afraid I’m one of those 00s developers and can relate. Back then many startups were being launched on super simple stacks. With all of that complexity/word salad from TFA, where’s the value delivered? Presumably there’s a product somewhere under all that infrastructure, but damn, what’s left to spend on it after all the infrastructure variable costs? I get it’s a list of preferences, but still once you’v…

That's for slower projects. You know the old adage "fast, cheap, good: pick two"? With startups, you're forced to pick fast. You're still probably not gonna make it, but if you don't build fast, you definitely won't.

"That's what they want you to think"
Post reply on HN