Live data from Hacker News

Cutting down AWS cost by $150k per year simply by shutting things off

tuananh.net

71–80 of 194 posts

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#71
I recently helped save $150k per year by deleting node_modules.

I noticed that one of our S3 buckets had high data transfer costs, a bucket that our app downloads HTML+JS assets from when we push out a new release. I downloaded the "directory" of files for our latest release and saw it was mostly node_modules. I checked the code and confirmed that, yes, if this file exists in the bucket then it'll be downloaded by the user. I wrote a quick Python script to list out each directory that had this problem, and a quick Slack message to the appropriate team later, we discovered the specific commit that was the cause, a change to our CI that inadvertently uploaded that directory when we wanted to ignore it.

A few months later, I checked the billing metrics, the effect was an avg of $12,500 reduction in cost for this bucket, or around $150k per year, or 4% of our bill. Not bad for one hour of work. Over the course of a quarter I reduced our bill by over $1m, or around 30% of our bill.

I might write a blog post explaining how to go about something like that. A lot of people are not familiar with tools like Trusted Advisor which can easily tell you if you have, for example, unused EC2 instances that can be terminated.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#72
> The best optimization is simply shutting things off

This is the way.

A similar idea has been bouncing around in my mind for a while now. An ideal, turnkey system would do the following:

- Execute via Lambda (serverless).

- Support automated startup and shutdown of various AWS resources on a schedule influenced by specially formatted tags.

- Enable resources to be brought back up out of schedule when demand dictates.

- Operate as a TCP/HTTP proxy that can delay clients so that a given service can be started when it is dormant or, even better, the service isn't serverless but you want it to be. This can't work for everything, but perhaps enough things such that the need to run always on services is reduced.

Cloud Custodian [1] can purportedly do some of this, but I've been reluctant to learn yet another YAML-based DSL to use it.

So this is my "make things designed to be always-on serverless instead" project and the work AWS has done to make Java apps function on Lambda keeps me thinking about the potential to take things that 1) have a relatively long startup time and 2) are designed to be long running service loops, and find a way to force them into the serverless execution model.

[1] https://cloudcustodian.io/

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#73

It is interesting to note that the author works at VPBank, which is one of the larger Vietnamese banks. Saving $150k per year on an AWS bill, is really nothing to them. The fact that they even outsource their compute to AWS is kind of surprising when they could just fill up their existing data centers (like VNTT https://vntt.com.vn/ ) with equipment, and save a whole lot more money.

And it's also interesting that they can outsource their compute to AWS because AWS's nearest data centers are in Hong Kong & Singapore. I didn't realize a bank would allow that.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#74
post #24

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

The huge achievement of Amazon was designing a system and selling it to people where developers no longer had to pre-approve spending. Previously developers were hamstrung by purchase order requirements; it could take weeks to authorize a single computer. Now the pendulum has swung in the direction. Developers can spend unlimited amounts of company money without realizing, billed in arrears. And in many cases this is…

Overall good points but don't forget that pre-approval processes resulted in asking for resources that exceeded the near term needs and once approved ongoing costs were rarely fully reviewed. I have personal experience with "enterprise" clients making a huge months long process to get server resources, reminding us that changes would take 30+ days. when the project was over and we did everything we could to let them know that the servers could be spun down or put to other uses we got back a "ok thanks!" only to find them still running our project code YEARS later. This is infra that was costing them about 1 engineer FTE per year, not even a 10$/mo toy env

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#75
post #71

I recently helped save $150k per year by deleting node_modules. I noticed that one of our S3 buckets had high data transfer costs, a bucket that our app downloads HTML+JS assets from when we push out a new release. I downloaded the "directory" of files for our latest release and saw it was mostly node_modules. I checked the code and confirmed that, yes, if this file exists in the bucket then it'll be downloaded by th…

please do :) i would love to learn more about this

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#76
post #40

Earlier quoted context omitted.

I think "engineer" isn't really the correct word to use for the artisans who build much of the tooling used by most companies. An engineer either wears a striped hat and drives a train, or, went to a credentialed school and passed a bunch of test and is allowed to sign documents that state "this thing, if built this way, won't collapse and kill people." It is expected that an engineer can predict with reasonable accu…

“engineers stick to doing things that are well understood and predictable” I’m calling BS on this. If this were true, we’d still be a ground species. Engineering has been and will always be about creating something electrical, mechanical, computerized, or all, that solves a problem. Understood or not. Engineers are not oracles. They can not predict whether a tower built in Italy will eventually begin to lean due to e…

“Engineering is the art of modelling materials we do not wholly understand, into shapes we cannot precisely analyse so as to withstand forces we cannot properly assess, in such a way that the public has no reason to suspect the extent of our ignorance.” - Dr. AR Dykes

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#77
Readers may find Steampipe's [1] AWS Thrifty Mod [2] useful. It will automatically scan multiple accounts and regions for 50 cost saving opportunities - many of which are looking for over-provisioned or unused resources. For example, it's crazy how much you can save by doing things like just converting your EBS volumes to the newer gp3 type. Combine with Flowpipe [3] to automate checks and actions. It's all open source and extensible.

1 - https://github.com/turbot/steampipe 2 - https://github.com/turbot/steampipe-mod-aws-thrifty 3 - https://github.com/turbot/flowpipe

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#78

> You go home. So you shut things down. Sorry for the rant, but this is usually wrong. The amount of people that just keeps their computer on is noticeable. And when I ask it's usually "just to avoid having to wait" or "I've always done that". I personally always hibernate my computer. When I turn it off it takes more time, but I'm already on the other side of the building so I don't care. When I turn it on it takes…

I keep mine on because it's my jumphost for working remotely. But I agree many people don't need to do this. My company, though, wants people to leave their PCs on so they can get automatic updates and be centrally managed.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#79

Earlier quoted context omitted.

“engineers stick to doing things that are well understood and predictable” I’m calling BS on this. If this were true, we’d still be a ground species. Engineering has been and will always be about creating something electrical, mechanical, computerized, or all, that solves a problem. Understood or not. Engineers are not oracles. They can not predict whether a tower built in Italy will eventually begin to lean due to e…

“Engineering is the art of modelling materials we do not wholly understand, into shapes we cannot precisely analyse so as to withstand forces we cannot properly assess, in such a way that the public has no reason to suspect the extent of our ignorance.” - Dr. AR Dykes

Ah the eloquence of Dr. AR Dykes, perfectly said. Thank you.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#80
post #24

Earlier quoted context omitted.

The huge achievement of Amazon was designing a system and selling it to people where developers no longer had to pre-approve spending. Previously developers were hamstrung by purchase order requirements; it could take weeks to authorize a single computer. Now the pendulum has swung in the direction. Developers can spend unlimited amounts of company money without realizing, billed in arrears. And in many cases this is…

We should be getting kickbacks from Amazon for all the work we’ve done for their bottom line.

We kinda do; Amazon puts upwards pressure on US engineering salaries.
Post reply on HN