Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost every infrastructure decision I endorse or regret

cep.dev

Post reply on HN
Print view View original

101–110 of 644 posts

Re: Almost every infrastructure decision I endorse or regret

#101

Post by leetharris »

calvinmorrison wrote: post #42
sgarland wrote: post #15

Earlier quoted context omitted.

The downside is then you have many, many DBs to fight with, to monitor, to tune, etc. This is rarely a problem when things are small, but as they grow, the bad schema decisions made by empowering DBA-less teams to run their own infra become glaringly obvious.

It's because I hate databases and programming separately. I would rather slow code then have to dig into some database procdure. Its just another level of separation thats too mentally hard to manage. Its like... my queries go into a VM and now I have to worry about how the VM is performing. I wish and maybe there is a programming language with first class database support. I mean really first class not just let me r…

The closest thing to what you're describing is Prisma in Node. It generates a Typescript file from your schema so you get code completion on your data. And it exists somewhere between a query builder and a traditional ORM.

I have worked in many languages with many ORMs and this has been my personal favorite.

Re: Almost every infrastructure decision I endorse or regret

#102

Post by icameron »

LispSporks22 wrote: post #74

I feel like this is overkill for a startup. Why not dump your application server and dependencies into rented data center (or EC2 if you must) and setup a coarse DR? Maybe start with a monolith in PHP or Rails. None of that word salad sounds like startup to me, but then again everyone loves to refer to themselves as a startup (must be a recruiting tool?), so perhaps muh dude is spot on.

I would like to know what you’re being downvoted for. It’s not bad advice, necessarily… this was the way 20 years ago. I mean isn’t hacker news running kind of like this as a monolith on a single server? People might be surprised how far you can get with a simple setup.

Re: Almost every infrastructure decision I endorse or regret

#103

Post by maccard »

sroussey wrote: post #51

If you are startup that can can’t afford a DBA, then why why why are you using Kubernetes?

Because I can go from main.go to a load balanced, autoscaling app with rolling deploys, segeregated environments, logging & monitoring in about 30 minutes, and never need to touch _any_ of that again. Plus, if I leave, the guy who comes after me can look at a helm chart, terraform module + pipeline.yml and figure out how it works. Meanwhile, our janq shell script based task scheduler craps out on something new every month. What started as 15 lines of "docker run X, sleep 30 docker kill x" is now a polyglot monster to handle all sorts of edge cases.

I have spent vanishingly close to 0 hours on maintaining our (managed) kubernetes clusters in work over the past 3 years, and if I didn't show up tomorrow my replacement would be fine.

Re: Almost every infrastructure decision I endorse or regret

#105

Post by marcosdumay »

el_benhameen wrote: post #59
Scubabear68 wrote: post #3

The kitchen sink database used by everybody is such a common problem, yet it is repeated over and over again. If you grow it becomes significant tech debt and a performance bottleneck. Fortunately, with managed DBs like RDS it is really easy to run individual DB clusters per major app.

Lots of interesting comments on this one. Anyone have any good resources for learning how not to fuck up schema/db design for those of us who will probably never have a DBA on the team?

> not to fuck up schema/db design

The neat thing is, you don't. Nobody ever avoids fucking up db design.

The best you can do is decide what is really important to get right, and not fuck that part up.

Re: Almost every infrastructure decision I endorse or regret

#106

Post by Scubabear68 »

electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

Elsewhere today I recommended RDS, but was thinking of small startup cases that may lack infrastructure chops.

But you are totally right it can be expensive. I worked with a startup that had some inefficient queries, normally it would matter, but with RDS it cost $3,000 a month for a tiny user base and not that much data (millions of rows at most).

Re: Almost every infrastructure decision I endorse or regret

#107

Post by amluto »

steveBK123 wrote: post #11
electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

RDS pricing is deranged at the scales I've seen too. $60k/year for something I could run on just a slice of one of my on-prem $20k servers. This is something we would have run 10s of. $600k/year operational against sub-$100k capital cost pays DBAs, backups, etc with money to spare. Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bullet…

I have a small MySQL database that’s rather important, and RDS was a complete failure.

It would have cost a negligible amount. But the sheer amount of time I wasted before I gave up was honestly quite surprising. Let’s see:

- I wanted one simple extension. I could have compromised on this, but getting it to work on RDS was a nonstarter.

- I wanted RDS to _import the data_. Nope, RDS isn’t “SUPER,” so it rejects a bunch of stuff that mysqldump emits. Hacking around it with sed was not confidence-inspiring.

- The database uses GTIDs and needed to maintain replication to a non-AWS system. RDS nominally supports GTID, but the documented way to enable it at import time strongly suggests that whoever wrote the docs doesn’t actually understand the purpose of GTID, and it wasn’t clear that RDS could do it right. At least Azure’s docs suggested that I could have written code to target some strange APIs to program the thing correctly.

Time wasted: a surprising number of hours. I’d rather give someone a bit of money to manage the thing, but it’s still on a combination of plain cloud servers and bare metal. Oh well.

Re: Almost every infrastructure decision I endorse or regret

#108

Post by ChuckMcM »

This is fabulous. I keep lists like this in my notebook(s). The critical thing here is that you shouldn't dwell on your "wrong" choices, instead document the choice, what you thought you were getting, what you got, and what information would have been helpful to know at the time of decision (or which information you should have given more weight at the time of the decision.) If you do this, you will consistently get better and better.

And by far "automate all the things" is probably my number one suggestion for DevOps folks. Something that saves you 10 minutes a day pays for itself in a month when you have a couple of hours available to diagnose and fix a bug that just showed up. (5 days a week X 4 weeks X 10 minutes = 200 minutes) The exponential effect of not having to do something is much larger than most people internalize (they will say, "This just takes me a couple of minutes to do." when in fact it takes 20 to 30 minutes to do and they have to do it repeatedly.)

Re: Almost every infrastructure decision I endorse or regret

#109

Post by marcosdumay »

inquist wrote: post #88
sgarland wrote: post #37

Earlier quoted context omitted.

But then the DB Team – if you have one – is responsible for 50 databases, each full of their own unique problems. This will undoubtedly go over poorly, but honestly I think every data decision should be gated through the DB Team (again, if you have them). Your proposed schema isn’t normalized? Straight to jail. You don’t want to learn SQL? Also straight to jail. You want to use a UUIDv4 as a primary key? Believe it o…

What’s wrong with uuidv4 as PK?

Serial integers always work better than any uuid as PKs, but the thing with uuid4 is that it disrupts any kind of index or physical ordering you decide to put on your data.

Uuids are really for external communication, not in-system organization.

Re: Almost every infrastructure decision I endorse or regret

#110

Post by tomas789 »

endisneigh wrote: post #60

Great post. I do wonder - what are the simplest K8s alternatives? Many say in the database world, "use Postgres", or "use sqlite." Similarly there are those databases that are robust that no one has heard of, but are very limited like FoundationDB. Or things that are specialized and generally respected like Clickhouse. What are the equivalents of above for Kubernetes?

You can always use old boring AWS EC2 and such. And sprinkle in some Terraform if you feel fancy. That would be my “use sqlite”

Kubernetes is probably “use postgres”

Post reply on HN