Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cutting down AWS cost by $150k per year simply by shutting things off

tuananh.net

Post reply on HN
Print view View original

91–100 of 194 posts

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#91

Post by latchkey »

JCharante wrote: post #73
latchkey wrote: post #15

It is interesting to note that the author works at VPBank, which is one of the larger Vietnamese banks. Saving $150k per year on an AWS bill, is really nothing to them. The fact that they even outsource their compute to AWS is kind of surprising when they could just fill up their existing data centers (like VNTT https://vntt.com.vn/ ) with equipment, and save a whole lot more money.

And it's also interesting that they can outsource their compute to AWS because AWS's nearest data centers are in Hong Kong & Singapore. I didn't realize a bank would allow that.

I thought it, but I wasn't going to say it. Vietnam's internet connection is notoriously unstable. The running joke is that sharks attack the fiber connections [0] because pointing fingers is a national past time. The fact that a major bank is relying on an external AWS like that, makes it even more comical.

My guess is that nobody in corporate approved this guys posting and if word got back, it would disappear quickly.

Reminds me to forward this to my buddies who run Timo, which VPBank used to own, but then dropped [1]. Timo was the first forward thinking bank in Vietnam with a great tech platform, likely because it was started and run by foreigners... ¯\_(ツ)_/¯.

[0] https://www.reddit.com/r/VietNam/comments/zvo553/sharks_ate_...

[1] https://fintechnews.sg/42738/vietnam/vietnams-challenger-tim...

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#92

Post by mytailorisrich »

pjc50 wrote: post #24
darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

The huge achievement of Amazon was designing a system and selling it to people where developers no longer had to pre-approve spending. Previously developers were hamstrung by purchase order requirements; it could take weeks to authorize a single computer. Now the pendulum has swung in the direction. Developers can spend unlimited amounts of company money without realizing, billed in arrears. And in many cases this is…

Yes but it is up to a company to control its spending. It must have a process and policy in place to deal with this. It's not Amazon's fault if it hasn't.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#95

Post by JohnMakin »

tuananh wrote: post #82
JohnMakin wrote: post #70

Earlier quoted context omitted.

Have you done this or attempted this yet? Every kubernetes cluster is different, but in my time working with them the last several years I anticipate the following issues: - dependent services not coming up in the order you expect/want - issues draining nodes due to crashlooping/erroring pods (can also be caused by dependent upstream services going down in wrong order) - Persistent Volume retention/synchronization -…

> I’ve had many managers over the years pitch this idea of “rapidly deployable/destructable EKS clusters” and the projects always get killed due to the complexity around this. This is exactly what we do: blue green eks cluster. We just thought if we do it on monthly basis, DRP will be piece of cake :)

Look forward to the writeup! thanks

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#96

Post by VoodooJuJu »

darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

And that's how insignificant the costs of cloud providers are in the grand scheme of things. It's a lot of money to a bootstrapping startup, but for the vast majority of these cloud providers' customers, it's a rounding error that's easily forgotten for 3 years.

And that's precisely why you and your little bootstrapper or indie firm should not be using globocloud: you do not have mountains of cash to piss away. Bare metal is trending again. And in this downturn, it's no wonder why. Smaller companies are getting smarter and more efficient. They've decided to chase money instead of cargo cults.

Globocorps burning cash on globocloud is not a signal for small fish to do the same - it's a signal to do the polar opposite. You're not going to become like them by copying what they're doing now. It will not work for you. Globocloud isn't successful because they shovel cash into AWS's shredder, they shovel cash into the shredder because they're successful.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#97

Post by ska »

tbalsam wrote: post #66
pyeri wrote: post #51

Earlier quoted context omitted.

That's the whole ploy with Agile, isn't it? In the classical SDLC or Waterfall paradigm, everything was pre-approved and signed off, not just the cost or billing but even the software design itself. Any change in the process and the designers had to raise a change request. Agile changed all of that and now we know how bad things can get with that.

No, Agile is about tight development loops. When weaponized/used by large corporations it often times turns into a Stay-Puft man of sorts, but as someone who _hates_ processes normally, I actually kinda like it and Kanban when done well. It basically helps keep things clean with decomposition and doesn't necessarily hamstring older devs as much while giving a good guideline for younger devs to work in. All things con…

> who _hates_ processes normally,

There is no such thing as "no process"; is something you always have whether you talk about it or not. The often heard "I hate process" is counterfactual then - what it really means is "I hate process that I see as intrusive/wasteful/whatever".

The +'ves you are listing are what comes from looking at how things are actually done and doing some of it a bit more thoughtfully.

The common -ves often come from stakeholders outside of developement injecting their needs ... sometimes this is unavoidable (e.g. regulatory) sometimes it is just political, but either way there are better and worse ways to do it.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#98

Post by ExoticPearTree »

cduzz wrote: post #83
reactordev wrote: post #63

Earlier quoted context omitted.

“engineers stick to doing things that are well understood and predictable” I’m calling BS on this. If this were true, we’d still be a ground species. Engineering has been and will always be about creating something electrical, mechanical, computerized, or all, that solves a problem. Understood or not. Engineers are not oracles. They can not predict whether a tower built in Italy will eventually begin to lean due to e…

Engineers manage risk and cost. They certainly make mistakes, like those couple buildings that are famously leaning over in SF and NYC, or the citycorp center where they got the wind sheer loads wrong and had to hot patch the building. But looking at the malarkey that goes on in "software engineering" or whatever -- clearly not engineering, at least not where I've seen it. Engineering: a process of repeatably solving…

> Engineering: a process of repeatably solving an understood problem predictably.

We call it help desk, not engineering.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#99

Post by happymellon »

malfist wrote: post #62
happymellon wrote: post #41

Earlier quoted context omitted.

I found that the problem happens mostly when companies 1. Don't ask developers how much something costs, engineers love optimisation, getting as much as possible out of a system for cheap is great fun. 2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in…

Lots of that is just bad design by going the easiest route. It's easy not to grant engineers access to the billing dashboard. It's easy to put everything in the same aws account. Inside Amazon, we're supposed to set up new aws accounts for every service and realm, so we know how much X service's beta environment is costing

Indeed, you need to share resources?

Plenty of ways of doing that, like making a cross account shared VPC for example.

Everything is still accountable.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#100

Post by tehlike »

bdcravens wrote: post #64
tehlike wrote: post #32

Cutting down AWS cost by 90% by simply moving to hetzner.

Was doing some research this weekend on cloud exit. Hetzner is attractive, but our company is pretty much limited to the US (no international companies due to our current business model). How practical would it be? Also, I've seen a lot of concern over blocked IPs, especially for lower-cost hosts. Is that an issue with Hetzner?

Hetzner cloud supports us-west and us-east.
Post reply on HN