Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cutting down AWS cost by $150k per year simply by shutting things off

tuananh.net

Post reply on HN
Print view View original

141–150 of 194 posts

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#141

Post by gottorf »

cduzz wrote: post #83
reactordev wrote: post #63

Earlier quoted context omitted.

“engineers stick to doing things that are well understood and predictable” I’m calling BS on this. If this were true, we’d still be a ground species. Engineering has been and will always be about creating something electrical, mechanical, computerized, or all, that solves a problem. Understood or not. Engineers are not oracles. They can not predict whether a tower built in Italy will eventually begin to lean due to e…

Engineers manage risk and cost. They certainly make mistakes, like those couple buildings that are famously leaning over in SF and NYC, or the citycorp center where they got the wind sheer loads wrong and had to hot patch the building. But looking at the malarkey that goes on in "software engineering" or whatever -- clearly not engineering, at least not where I've seen it. Engineering: a process of repeatably solving…

> Engineers manage risk and cost.

"Any idiot can build a bridge that stands, but it takes an engineer to build a bridge that barely stands."

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#142

Post by killingtime74 »

devin wrote: post #131
paxys wrote: post #125

I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the…

Unfortunately it seems like businesses only wake up and asks for cost reduction on the infrastructure spend side once the problem is out of control. At that point, the level of operations effort to get it under control feels more like a "big rewrite" than a collection of small tweaks.

It's opportunity cost. If it's not a problem then you can just make more money with your time.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#143

Post by StratusBen »

pizzafeelsright wrote: post #104
StratusBen wrote: post #89

Disclosure: I'm CEO of https://www.vantage.sh/ -- a cloud cost observability platform. I previously worked at both AWS and DigitalOcean. For people looking for how to save money on AWS - I'd [selfishly] recommend connecting up to Vantage. We profile AWS for all sorts of savings and give you the information on how much we can save prior to you paying us. It can be a good gut-check if nothing else on how well optimized…

Silly request. The amount of paperwork, lawyers, and time required to connect our service makes it impossible to validate savings. Is there an offline method? I have not looked at vantage to see if it's possible.

Unfortunately we don't have an option for that route -- but we're happy to help support any paperwork for getting things up and running if you contact me or my team: ben [at] vantage [dot] sh.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#144

Post by deepsun »

mullingitover wrote: post #134
paxys wrote: post #125

I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the…

If your company has a spending commitment with AWS in order to get a few percent savings, and it's just barely hitting the contracted amount, it may not be worth the effort to pursue any cost savings. Suppose your company has committed to hit 5M in spend, and they're just barely inching over the line at 5.01M. You might spend a bunch of time and labor expense knocking it down to 4M of usage and not really move the bo…

Yep, that's how AWS sales make money.

Theoretically, your company could reduce their commitment to 4M next year, but the AWS sales would start negotiating hard against it, like "you will not get the same discount with less commitment".

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#146

Post by stevejb »

thefourthchime wrote: post #4

I've mentioned this before, in my company (big media company) I saw some S3 costs creeping up each month. I looked into it and it was a system we abandoned that was still copying files to this bucket. I reached out to the team and they turned it off, it saved us $1m a year. The higher-ups rewarded me by telling me that a team should have caught this so I should meet with them now.

I work for a company called CloudFix, and we are solely focused on AWS costs. We do automated AWS cost optimization. We find one of two reactions when we deliver savings to customers:

(A) "Hey wow, this is great! We are so excited to be saving from here on out." OR, (B) "This should have been caught earlier. $TEAM was supposed to be experts..." and then blame game starts.

It is really unfortunate when institutions react in the latter way. Often the engineers are assigned to cost optimization, along with a million other things. And, the incentives aren't really aligned well to reward savings. For example, S3 Intelligent Tiering is the right thing in 99.9% of cases - so it should be your default bucket type. BUT, engineers often face only downside risk for the change, and very little upside reward. And, it isn't their money so they just leave it. The cost of overprovisioned S3 can be staggering!

What is really needed is to establish a proper FinOps discipline, put someone in charge of cost savings, and make sure incentives are aligned properly. And of course check out CloudFix if you can!

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#147

Post by mk89 »

mullingitover wrote: post #134
paxys wrote: post #125

I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the…

If your company has a spending commitment with AWS in order to get a few percent savings, and it's just barely hitting the contracted amount, it may not be worth the effort to pursue any cost savings. Suppose your company has committed to hit 5M in spend, and they're just barely inching over the line at 5.01M. You might spend a bunch of time and labor expense knocking it down to 4M of usage and not really move the bo…

I agree with this comment.

Ironically, I was asked by a manager some time ago if we can imagine using some (more) resources from AWS to reach the next spending commitment. If you're just below the threshold, it's probably inconvenient.

EDIT: To give some context: you can only do this, of course, when you know that you're not really wasting resources, otherwise you end up with just burning money to save little to no money :)

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#148

Post by robocat »

cduzz wrote: post #40
darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

I think "engineer" isn't really the correct word to use for the artisans who build much of the tooling used by most companies. An engineer either wears a striped hat and drives a train, or, went to a credentialed school and passed a bunch of test and is allowed to sign documents that state "this thing, if built this way, won't collapse and kill people." It is expected that an engineer can predict with reasonable accu…

> An engineer [] went to a credentialed school and passed a bunch of test and is allowed to sign documents that state "this thing, if built this way, won't collapse and kill people."

Ahhh - that old craptacular definition. You completely ignore mechanical engineers, chemical engineers, electrical & electronics engineers. Not all engineers make bridges.

Secondly, the implied cause and effect even within civil engineering is a fantasy. Signatures on documents by credentialed engineers doesn't prevent disasters as you noted: Bridges fall down, buildings burn. Read the engineering reports on civil engineering disasters, and look at the consequences for the engineers involved.

You do some handwaving about organizational/process problems, but actually that is the key to safe engineering. Organisations deliver engineering projects and they do it across jurisdictional borders using insurance and liability and with a variety of other means that work: "signatures don't prevent disasters".

Lockheed Martin's skunk-works and SpaceX are real engineering. Any good definition of engineering needs to encompass an extremely wide variety of activities.

Engineering is compromise. I have no love for Musk but him saying build that actuator for less than $5k is actually true engineering: https://news.ycombinator.com/item?id=39085892

I would like to know the psychology behind why people wish to believe credentialed signatures are so powerful? Maybe a cross between two concepts #1: "that individual engineers run the world" and #2: "that retributive punishment of individuals works as a deterrent". I think concept #1 comes from the egotist idea of most engineer-types that we are the center of everything (I need a whole article to explain the concept). I think concept #2 is related to beliefs about the value of incarceration and also punishment beliefs derived from religion (especially in the USA where prisons are not fixing problems?).

Edit: issue #3: the idea that we should make rules about what words mean. It takes a certain worldview to think words should be defined rather than evolve (or worse that words should be part of a justice system)

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#149

Post by stusmall »

octopoc wrote: post #120

Here's a startup idea: a profiler for infrastructure-as-code that shows how much each line of code is costing per month, instead of showing where the CPU spends most of its time.

Have you looked at OpenCost? It's more k8s focused but of a similar idea

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#150

Post by caeril »

mullingitover wrote: post #134
paxys wrote: post #125

I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the…

If your company has a spending commitment with AWS in order to get a few percent savings, and it's just barely hitting the contracted amount, it may not be worth the effort to pursue any cost savings. Suppose your company has committed to hit 5M in spend, and they're just barely inching over the line at 5.01M. You might spend a bunch of time and labor expense knocking it down to 4M of usage and not really move the bo…

Spending commitment on AWS? Are you referring to ec2 reserved instances?

Even if you have one year reservations on your instances, starting service migrations/deprecations now would pay off quickly enough. Your commitment expires in 6 months, on an average basis.

Post reply on HN