Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost every infrastructure decision I endorse or regret

cep.dev

Post reply on HN
Print view View original

191–200 of 644 posts

Re: Almost every infrastructure decision I endorse or regret

#191

Post by Cacti »

bigstrat2003 wrote: post #164
jacurtis wrote: post #146

Earlier quoted context omitted.

I mostly agreed with OP's article, but you basically nailed all of the points of disagreement I did have. Jira: Its overhyped and overpriced. Most HATE jira. I guess I don't care enough. I've never met a ticket system that I loved. Jira is fine. Its overly complex sure. But once you set it up, you don't need to change it very often. I don't love it, I don't hate it. No one ever got fired for choosing Jira, so it gets…

I'm right there with you on Jira. The haters are wrong - it's a decent enough ticket system, no worse than anything else I've used. You can definitely torture Jira into something horrible, but that's not Jira's fault. Bad managers will ruin any ticket system if they have the customization tools to do so.

Using Jira feels like using IBM enterprise web software from 2005, and I am simply not going to make my teams put up with that amount of inanity.

Re: Almost every infrastructure decision I endorse or regret

#192

Post by nyc_data_geek »

stingraycharles wrote: post #173
nyc_data_geek wrote: post #86

Earlier quoted context omitted.

Some orgs are looking at moving back to on prem because they're figuring this out. For a while it was vogue to go from capex to opex costs, and C suite people were incentivized to do that via comp structures, hence "digital transformation" ie: migration to public cloud infrastructure. Now, those same orgs are realizing that renting computers actually costs more than owning them, when you're utilizing them to a signif…

That’s made possible because of all the orchestration platforms such as Kubernetes being standardized, and as such you can get pretty close to a cloud experience while having all your infrastructure on-premise.

Yes, virtualization, overprovisioning and containerization have all played a role in allowing for efficient enough utilization of owned assets that the economics of cloud are perhaps no longer as attractive as they once were.

Re: Almost every infrastructure decision I endorse or regret

#193

Post by latch »

theptip wrote: post #162
tw04 wrote: post #130

Earlier quoted context omitted.

>By the time your db is big enough to cost an unreasonable amount on RDS, you’re likely a big enough team to have options. By the time your db is big enough to cost an unreasonable amount on RDS, you've likely got so much momentum that getting off is nearly impossible as you bleed cash. You can buy a used server and find colocation space and still be pennies on the dollar for even the smallest database. If you're doi…

That’s just another way of saying the opportunity cost isn’t worth paying to do the migration. Optionality and flexibility are extremely valuable, and that is why cloud compute continues to be popular, especially for rapidly/burstily growing businesses like startups.

I don't mean to pick on your specific comments, but I find these analysis almost always lack a crucial perspective: level of knowledge. This is the single biggest factor, and it's the hardest one to be honest about. No one wants to say "RDS is a good choice . . . because I don't know how nor have I ever self managed a database."

If you want a different opportunity cost, get people with different experience. If RDS is objectively expensive, objectively slow, but subjectively easy, change the subject.

Re: Almost every infrastructure decision I endorse or regret

#195

Post by macNchz »

tw04 wrote: post #130
macNchz wrote: post #31

Earlier quoted context omitted.

> Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bulletproof.. it makes sense? But how many cases are there of this? Most software startups these days? The blog post is about work done at a startup after all. By the time your db is big enough to cost an unreasonable amount on RDS, you’re likely a big enough team to have options. If yo…

>By the time your db is big enough to cost an unreasonable amount on RDS, you’re likely a big enough team to have options. By the time your db is big enough to cost an unreasonable amount on RDS, you've likely got so much momentum that getting off is nearly impossible as you bleed cash. You can buy a used server and find colocation space and still be pennies on the dollar for even the smallest database. If you're doi…

In the small SaaS startup case, I’d say the production database is typically the most critical single piece of infra, so self hosting is just not a compelling proposition unless you have a strong technical reason where having super powerful database hardware is important, or a team with multiple people who have sysadmin or DBA experience. I think both of those cases are unusual.

I’ve been the guy managing a critical self-hosted database in a small team, and it’s such a distraction from focusing on the actual core product.

To me, the cost of RDS covers tons of risks and time sinks: having to document the db server setup so I’m not the only one on the team who actually knows how to operate it, setting up monitoring, foolproof backups so I don’t need to worry that they’re silently failing because a volume is full and I misconfigured the monitoring, PITR for when someone ships a bad migration, one click HA so the database itself is very unlikely to wake me at 3am, blue/green deploys to make major version upgrades totally painless, never having to think about hardware failures or borked dist-upgrades, and so on.

Each of those is ultimately either undifferentiated work to develop in-house RDS features that could have been better spent on product, or a risk of significant data loss, downtime, or firefighting. RDS looks like a pretty good deal, up to a point.

Re: Almost every infrastructure decision I endorse or regret

#197

Post by neeleshs »

steveBK123 wrote: post #11
electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

RDS pricing is deranged at the scales I've seen too. $60k/year for something I could run on just a slice of one of my on-prem $20k servers. This is something we would have run 10s of. $600k/year operational against sub-$100k capital cost pays DBAs, backups, etc with money to spare. Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bullet…

Out of curiosity, who is your onprem provider?

Re: Almost every infrastructure decision I endorse or regret

#198

Post by mise_en_place »

cedws wrote: post #91
hintymad wrote: post #73

> EKS My contrarian view is that EC2 + ASG is so pleasant to use. It’s just conceptually simple: I launch an image into an ASG, and configure my autoscale policies. There are very few things to worry about. On the other hand, using k8s has always been a big deal. We built a whole team to manage k8s. We introduce dozens of concepts of k8s or spend person-years on “platform engineering” to hide k8s concepts. We publish…

K8S is a disastrous complexity bomb. You need millions upon millions of lines of code just to build a usable platform. Securing Kubernetes is a nightmare. And lock-in never really went away because it's all coupled with cloud specific stuff anyway. Many of the core concepts of Kubernetes should be taken to build a new alternative without all the footguns. Security should be baked in, not an afterthought when you need…

kubeadm + fabric + helm got me 99% of the way there. My direct report, a junior engineer, wrote the entire helm chart from our docker-compose. It will not entirely replace our remote environment but it is nice to have something in between our SDK and remote deployed infra. Not sure what you meant by security; could you elaborate? I just needed to expose one port to the public internet.

Re: Almost every infrastructure decision I endorse or regret

#199

Post by osigurdson »

electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

Cloud was supposed to be a commodity. Instead it is priced like at burger at the ski hill.

Re: Almost every infrastructure decision I endorse or regret

#200

Post by webo »

eduction wrote: post #149
dalyons wrote: post #121

Earlier quoted context omitted.

Why would you make it a shared resource if you don’t have to? Decades of experience have shown us the massive costs of doing so - the crippled velocity and soul crushing agony of dba change control teams, the overhead salary of database priests, the arcane performance nightmares, the nuclear blast radius, the fundamental organizational counter-incentives of a shared resource . Why on earth would we choose to pay thos…

You wouldn’t but in any decent sized organization you will have to. If it is an organization that needs to exist there will be some common set of critical data.

In my experience, isolated (repeated) data storage paradigm is even more common at large organizations. They share data via services, ETLs, event buses, etc.
Post reply on HN