Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost every infrastructure decision I endorse or regret

cep.dev

Post reply on HN
Print view View original

391–400 of 644 posts

Re: Almost every infrastructure decision I endorse or regret

#391

Post by graemep »

dfgdfg34545456 wrote: post #364
electroly wrote: post #6

> The markup cost of using RDS (or any managed database) is worth it. Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a…

For big companies such as banks this cost comparison is not as straight forward. They have whole data centres just sitting there for disaster recovery. They periodically do switchovers to test DR. All of this expense goes away when they migrate to cloud.

> All of this expense goes away when they migrate to cloud.

They need to replicate everything in multiple availability zones, which is going to be more expensive than replicating data centres.

They still need to test their cloud infrastracuture works.

Re: Almost every infrastructure decision I endorse or regret

#392

Post by jq-r »

morsecodist wrote: post #131

> Picking AWS over Google Cloud I know this is an unpopular opinion but I think google cloud is amazing compared to AWS. I use google cloud run and it works like a dream. I have never found an easier way to get a docker container running in the cloud. The services all have sensible names, there are fewer more important services compared to the mess of AWS services, and the UI is more intuitive. The only downside I ha…

> "regular cadence meetings with our AWS account manager" and I am not sure what could be discusse.

As being on a number of those calls, its just a bunch of crap where they talk like a scripted bot reading from corporate buzzword bingo card over a slideshow. Their real intention is two fold. To sell you even more AWS complexity/services, and to provide "value" to their person of contact (which is person working in your company).

We're paying north of 500K per year in AWS support (which is a highway robbery), and in return you get a "team" of people supposedly dedicated to you, which sounds good in theory but you get a labirinth of irresponsiblity, stalling and frustration in reality.

So even when you want to reach out to that team you have to first to through L1 support which I'm sure will be replaced by bots soon (and no value will be lost) which is useful in 1 out of 10 cases. Then if you're not satisfied with L1's answer(s), then you try to escalate to your "dedicated" support team, then they schedule a call in three days time, or if that is around Friday, that means Monday etc.

Their goal is to stall so you figure and fix stuff on your own so they shield their own better quality teams. No wonder our top engineers just left all AWS communication and in cases where unavoidable they delegate this to junior people who still think they are getting something in return.

Re: Almost every infrastructure decision I endorse or regret

#393

Post by gregw2 »

rswail wrote: post #247
steveBK123 wrote: post #66

Earlier quoted context omitted.

Once you commit to more deeply Amazon flavored parts of AWS like Aurora, aren't you now fairly committed to hoping your scale never exceeds the cost-benefit tradeoff?

Aurora supports standard Postgres clients. So moving to/from Aurora/RDS/own EC2/on-prem should be a matter of networking and changing connection strings in the clients. Your operational requirements and processes (backup/restore, failover, DR etc) will change, but that's because you're making a deliberate decision weighing up those costs vs benefits.

Pro tip side note:

You can use DNS to mitigate the pain of changing those connection strings, decoupling client change management from backend change process, or if you had foresight, not having to change client connection strings at all.

Re: Almost every infrastructure decision I endorse or regret

#394

Post by ildjarn »

Reading this I couldn’t help but think: yeah all of these points make sense in isolation, but if you look at the big picture, this is an absurd level of complexity.

Why do we need entire teams making 1000s of micro decisions to deploy our app?

I’m hungry for a simpler way, and I doubt I’m alone in this.

Re: Almost every infrastructure decision I endorse or regret

#395

Post by marcinzm »

wodenokoto wrote: post #231
morsecodist wrote: post #131

> Picking AWS over Google Cloud I know this is an unpopular opinion but I think google cloud is amazing compared to AWS. I use google cloud run and it works like a dream. I have never found an easier way to get a docker container running in the cloud. The services all have sensible names, there are fewer more important services compared to the mess of AWS services, and the UI is more intuitive. The only downside I ha…

If you are big enough to have regular meetings with AWS you are big enough to have meetings with GCP. I’ve had technicians at both GCP and Azure debug code and spend hours on developing services.

> I’ve had technicians at both GCP and Azure debug code and spend hours on developing services.

Almost every time Google pulled in a specialist engineer working on a service/product we had issues with it was very very clear the engineer had no desire to be on that call or to help us. In other words they'd get no benefit from helping us and it was taking away from things that would help their career at Google. Sometimes they didn't even show up to the first call and only did to the second after an escalation up the management chain.

Re: Almost every infrastructure decision I endorse or regret

#396

Post by dfgdfg34545456 »

nightfly wrote: post #372
dfgdfg34545456 wrote: post #364

Earlier quoted context omitted.

For big companies such as banks this cost comparison is not as straight forward. They have whole data centres just sitting there for disaster recovery. They periodically do switchovers to test DR. All of this expense goes away when they migrate to cloud.

> All of this expense goes away when they migrate to cloud. Just to pay someone else enough money to provide the same service and make a profit while do it

Well corporations pay printers to do their printing because they don't want to be in the business of printing. It's the same with infrastructure, a lot of corporations simply don't want to be in the data centre business.

Re: Almost every infrastructure decision I endorse or regret

#397

Post by marcinzm »

davedx wrote: post #351
isbvhodnvemrwvn wrote: post #310

Earlier quoted context omitted.

Setting up a multi-az db with automatic failover, incremental backups and PiTR, automated runbooks and monitoring all that doesn't take half a day, not even with RDS.

No, but again, that sounds like a lot of complexity your average startup does not need. Multi-az? Why?

Because their Enterprise client requires it on their due diligence paperwork.

Re: Almost every infrastructure decision I endorse or regret

#398

Post by marcinzm »

davedx wrote: post #349
ffsm8 wrote: post #313

Earlier quoted context omitted.

> Automating a database backup and testing it works takes half a day unless you’re doing something weird True story bro I'm sure that's possible if you're storing the backup on the same server you're restoring on and everything is on top of the line nvme storage. Otherwise your backup just started to run and will need another few days to finish. And that's only if you're running single master. You're massively undere…

I’ve implemented such a process for companies multiple times, bro. I know what I’m talking about.

And that's the problem. "It's easy for me because I've done it a dozen times so it's easy for everyone" is a very common fallacy.

Re: Almost every infrastructure decision I endorse or regret

#399

Post by nemothekid »

pclmulqdq wrote: post #186
nemothekid wrote: post #177

Earlier quoted context omitted.

If my scale exceeds the cost benefit tradeoff, then I will thank God/Allah/Buddah/Spaghetti Monster. These questions always sound flawed to me. It's like asking won't I regret moving to California and paying high taxes once I start making millions of dollars? Maybe? But that's an amazing problem to have and one that I may be much better equipped to solve. If you are small, RDS is much cheaper, and many company killin…

I assume that you do that math on all your new features too, right? The calculation of how much extra money they will bring in? On some level, AWS/GCP/California relies on you doing this calculation for the things that you can do it on easily (the savings of moving away), while not doing this calculation on things where it's hard to do (new development). That way, you can pretend that your new features are a lot more…

>The calculation of how much extra money they will bring in?

Yes, I've done the math. The piece you are missing is, saving money on infra will bring in $0 new dollars. There is a floor to how much money I can save. There is no ceiling to how much money the right feature can bring in. Penny pinching on infra, especially when the amount of money is saved is less than the cost of an engineer is almost always a waste of time while you are growing a company. If you are at the point where you are wasting 1x,2x,3x of an engineers salary of superflous infrastructure - then congratulations you have survived the great filter for 99% of startups.

>That way, you can pretend that your new features are a lot more valuable than the $Xk/year you will save by moving your infra.

Finding product market fit is 1000x harder than moving from RDS to On-prem. If you haven't solved PMF, then no amount of $Xk/year in savings will save you from having to shut down your company.

Re: Almost every infrastructure decision I endorse or regret

#400

Post by adrianmsmith »

hhsectech wrote: post #317
jumploops wrote: post #280

Earlier quoted context omitted.

Funny story time. I was once part of an acquisition from a much larger corporate entity. The new parent company was in the middle of a huge cloud migration, and as part of our integration into their org, we were required to migrate our services to the cloud. Our calculations said it would cost 3x as much to run our infra on the cloud. We pushed back, and were greenlit on creating a hybrid architecture that allowed us…

There are two possible scenarios here. Firstly, they can't find the talent to support what you implemented...or more likely, your docs suck! I've made a career out of inheriting other peoples whacky setups and supporting them (as well as fixing them) and almost always its documentation that has prevented the client getting anywhere. I personally dont care if the docs are crap because usually the first thing I do is u…

> Quite frequently though its outsourced development agencies that refuse to write it

It's also completely against their interest to write docs as it makes their replacement easier.

That's why you need someone competent on the buying side to insist on the docs.

A lot of companies outsource because they don't have this competency themselves. So it's inevitable that this sort of thing happens and companies get locked in and can't replace their contractors, because they don't have any docs.

Post reply on HN