Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost every infrastructure decision I endorse or regret

cep.dev

Post reply on HN
Print view View original

621–630 of 644 posts

Re: Almost every infrastructure decision I endorse or regret

#621

Post by cyounkins »

el_isma wrote: post #605
cyounkins wrote: post #122

I've climbed the mountain of learning the basics of kubernetes / EKS, and I'm thinking we're going to switch to ECS. Kubernetes is way too complicated for our needs. It wants to be in control and is hard to direct with eg CloudFormation. Load balancers are provisioned from the add-on, making it hard to reference them outside kubernetes. Logging on EKS Fargate to Cloudwatch appears broken, despite following the docs.…

I started with ECS (because I wanted to avoid the complexity of K8s) and regret it. I feel I wasted a lot of time there. In ECS, service updates would take 15 min or more (vs basically instant in K8s). ECS has weird limits on how many containers you can run on one instance [0]. And in the network mode where you can run more containers on a host, then the DNS is a mess (you need to lookup SRV records to find out the p…

Interesting that you say you worry about re-creating the cluster from scratch because I've experienced exactly the opposite. Our EKS cluster required so many operations outside CloudFormation to configure access control, add-ons, metrics server, ENABLE_PREFIX_DELEGATION, ENABLE_POD_ENI... It would be a huge risk to rebuild the EKS cluster. And applications hosted there are not independent because of these factors. It makes me very anxious working on the EKS cluster. Yes you can pay an extra $70/month to have a dev cluster, but it will never be equal to prod.

On the other hand, I was able to spin up an entire ECS cluster in a few minutes time with no manual operations and entirely within CloudFormation. ECS costs nothing extra, so creating multiple clusters is very reasonable, though separate clusters would impact packing efficiency. The applications can be fully independent.

> ECS has weird limits on how many containers you can run on one instance

Interesting. With ECS it says for c5.large the task limit is 2 with without trunking, 10 with.

With EKS

    $ ./max-pods-calculator.sh --instance-type c5.large --cni-version 1.12.6
    29
    $ ./max-pods-calculator.sh --instance-type c5.large --cni-version 1.12.6 --cni-prefix-delegation-enabled
    110

Re: Almost every infrastructure decision I endorse or regret

#622

Post by nescioquid »

> Multiple applications sharing a database [regret]

The industry has known this to be a stereotypically bad idea for generations now. It lead to things like the enterprise sevice bus, service-oriented architectures, and finally "micro services". Recently I've seen "micro services" that share the same database, so we've come full-circle.

Yet, every place I've worked was either laboring under a project to decouple two or more applications that were conjoined at the DB, or were still at the "this sucks but no one wants to fix it" stage.

How do we keep making this same mistake in industry?

Re: Almost every infrastructure decision I endorse or regret

#623

Post by el_isma »

cyounkins wrote: post #621
el_isma wrote: post #605

Earlier quoted context omitted.

I started with ECS (because I wanted to avoid the complexity of K8s) and regret it. I feel I wasted a lot of time there. In ECS, service updates would take 15 min or more (vs basically instant in K8s). ECS has weird limits on how many containers you can run on one instance [0]. And in the network mode where you can run more containers on a host, then the DNS is a mess (you need to lookup SRV records to find out the p…

Interesting that you say you worry about re-creating the cluster from scratch because I've experienced exactly the opposite. Our EKS cluster required so many operations outside CloudFormation to configure access control, add-ons, metrics server, ENABLE_PREFIX_DELEGATION, ENABLE_POD_ENI... It would be a huge risk to rebuild the EKS cluster. And applications hosted there are not independent because of these factors. It…

In ECS I had to recreate the cluster from scratch because some of the changes I wanted to do, CDK/CF wouldn't do.

My approach on Azure has been to rely as little as possible in their Infra-as-code, and do everything I can to setup the cluster using K8s native stuff. So, add-ons, RBAC, metrics, all I'd try to handle with Helm. That way if I ever need to change K8s provider, it "should" be easy.

Re: Almost every infrastructure decision I endorse or regret

#624

Post by Ir0nMan »

nyc_data_geek wrote: post #86
electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

Some orgs are looking at moving back to on prem because they're figuring this out. For a while it was vogue to go from capex to opex costs, and C suite people were incentivized to do that via comp structures, hence "digital transformation" ie: migration to public cloud infrastructure. Now, those same orgs are realizing that renting computers actually costs more than owning them, when you're utilizing them to a signif…

Yep. This.

Re: Almost every infrastructure decision I endorse or regret

#625

Post by pconstantine »

electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

People, who use MSFT SQL server in 2024 should suffer. For everybody else there's always Neon.

Re: Almost every infrastructure decision I endorse or regret

#626

Post by pconstantine »

illumin8 wrote: post #586
electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

This is because you are using SQL Server. Microsoft has intentionally made cloud pricing for SQL server prohibitively expensive for non-Azure cloud workloads by requiring per-core licensing that is extremely punitive for the way EC2 and RDS is architected. This has the effect of making RDS vastly more expensive than running the same workload on bare metal or Azure. Frankly, this is anti-competitive, and the FTC shoul…

Totally agree. It's cherry-picking some weird case that's not even close to a typical for startup.

Re: Almost every infrastructure decision I endorse or regret

#627

Post by asalahli »

marcinzm wrote: post #603
davedx wrote: post #578

Earlier quoted context omitted.

This is an oversimplification, but! Dumping postgres to a file is one command. scp the file to a different server is two commands. (Granted you need to setup ssh keys there too). I have implemented backups this way. With sqlite you only need the scp part. You can even push your backup file to an S3 bucket... with one command! Honestly, this argument mystifies me. Of course you can make it as complicated as you want t…

And that works right until you get to publish an incident report like this: https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-data...

> Our backups to S3 apparently don’t work either: the bucket is empty

It took them a data loss incident to find this out? This is just one of the many red flags mentioned in the article, IMO this incident isn't about relying on cloud backups vs self managing it

Re: Almost every infrastructure decision I endorse or regret

#628

Post by citrin_ru »

overstay8930 wrote: post #547
davedx wrote: post #267

Utter insanity. So much cost and complexity, and for what? Startups don’t think about costs or runway anymore, all they care about is “modern infrastructure”. The argument for RDS seems to be “we can’t automate backups”. What on earth?

Everyone who says they can run a database better than Amazon is probably lying or Has a story about how they had to miss a family event because of an outage. The point isn’t that you can’t do it, the point is that it’s less work for extremely high standards. It is not easy to configure multi region failover without an entire network team and database team unless you don’t give a shit about it actually working. Oh yea…

One don’t necessarily need to run a DB better than Amazon. Just sufficiently good for the product/service you’re are working on. And depending on specifics it may costs much less (but your mileage may vary).

Re: Almost every infrastructure decision I endorse or regret

#630

Post by compumike »

solatic wrote: post #593
compumike wrote: post #188

Earlier quoted context omitted.

Thanks for the feedback! We use iOS “Critical Alerts” and similar on Android that breaks through any Do-Not-Disturb settings. https://heiioncall.com/blog/better-alerting-for-heii-on-call... Would you be willing to give that a shot? It wakes me every time :) (It’s configurable too; we have vibrate-only or silenced modes. Think old-school beeper.) In the rare case that it doesn’t wake you, we have configurable escalati…

One of the things about phone calls is that they will continue to ring and ring until you actively decide to answer or ignore; it's not just a half-second "ping!" sound delivered once. This is also useful in meetings when everything else is closed up or silenced, to jar you out. I usually do not respond immediately to phone notifications, which I can handle async. Phone calls are by definition sync.

Ah yeah, these are no ordinary phone notifications. :)

Heii On-Call will keep alerting you with these “Critical Alerts” until you’ve manually acknowledged. (Or until it escalates to a teammate and they acknowledge…)

And at least on my phone they sound nothing like normal phone notifications, which I personally always have on vibrate and/or DND anyway.

Give it a try and I think you’ll like it.

Post reply on HN