Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cutting down AWS cost by $150k per year simply by shutting things off

tuananh.net

Post reply on HN
Print view View original

181–190 of 194 posts

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#181

Post by charlie0 »

TrianguloY wrote: post #39

> You go home. So you shut things down. Sorry for the rant, but this is usually wrong. The amount of people that just keeps their computer on is noticeable. And when I ask it's usually "just to avoid having to wait" or "I've always done that". I personally always hibernate my computer. When I turn it off it takes more time, but I'm already on the other side of the building so I don't care. When I turn it on it takes…

I always shutdown my machine at night and sometimes restart when I leave for lunch. I've noticed running Docker and other apps for a while makes my machine slower. I'm convinced there's a memory leak somewhere and restarting away fixes those issues.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#182

Post by echoangle »

mullingitover wrote: post #134
paxys wrote: post #125

I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the…

If your company has a spending commitment with AWS in order to get a few percent savings, and it's just barely hitting the contracted amount, it may not be worth the effort to pursue any cost savings. Suppose your company has committed to hit 5M in spend, and they're just barely inching over the line at 5.01M. You might spend a bunch of time and labor expense knocking it down to 4M of usage and not really move the bo…

Can you explain how this is perverse incentives? Wouldn’t this actually be a good way to align incentives of the employee and the company (it makes both of them want to save money)? Or do you mean it makes the employee increase cost unnecessarily from time to time so he can reduce it later?

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#183

Post by krageon »

pixl97 wrote: post #25
lnxg33k1 wrote: post #23

Earlier quoted context omitted.

I get paid the same whether or not I spend time going around to save company cost, it’s not like they’re going to share their savings, then shareholders get their juice, management gets their juice and Im the clown who went out of my way for them, who cares I do software not cost management

Which is why your slice of the organization gets a cloud budget. Don't keep your budget under control, well no bonuses for the employees. >Show me the incentive and I will show you the outcome

This is fun, but these policies are based on the fact that your managers don't care about you and in fact prefer you remain as poor as they can legally make you. So any kind of reasoning along these lines ("well this makes sense") just is not real and won't ever become real.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#184

Post by krageon »

gurchik wrote: post #166
Detrytus wrote: post #138

Earlier quoted context omitted.

Now, the inconvenient question: how much of that $1m savings ended up in your bank account as a bonus? Because certainly some of it should :-D

Not sure yet, but probably nothing. I completely understand the expectations written in this thread to receive something in return, but I've given this thought and I'm not sure how to do this in a fair way in this situation. First, I was given dedicated time that quarter to work on cost savings and other people weren't, if I received a bonus is that fair to other people who didn't have the same opportunity? Not to me…

You being at peace with "non-monetary benefits" is what depresses wages and real rewards for you, but more importantly all your colleagues.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#185

Post by throwaway883322 »

echoangle wrote: post #182
mullingitover wrote: post #134

Earlier quoted context omitted.

If your company has a spending commitment with AWS in order to get a few percent savings, and it's just barely hitting the contracted amount, it may not be worth the effort to pursue any cost savings. Suppose your company has committed to hit 5M in spend, and they're just barely inching over the line at 5.01M. You might spend a bunch of time and labor expense knocking it down to 4M of usage and not really move the bo…

Can you explain how this is perverse incentives? Wouldn’t this actually be a good way to align incentives of the employee and the company (it makes both of them want to save money)? Or do you mean it makes the employee increase cost unnecessarily from time to time so he can reduce it later?

I am not the parent poster, but I think your last sentence covers it well.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#186

Post by Marrand »

Salgat wrote: post #159
preommr wrote: post #126

Earlier quoted context omitted.

Something something snakes, something something unintended consequences.

Sabotaging your company's infrastructure for a bonus is a great way to end up in prison.

How would you prove intent to sabotage?

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#187

Post by pjc50 »

pyeri wrote: post #51
pjc50 wrote: post #24

Earlier quoted context omitted.

The huge achievement of Amazon was designing a system and selling it to people where developers no longer had to pre-approve spending. Previously developers were hamstrung by purchase order requirements; it could take weeks to authorize a single computer. Now the pendulum has swung in the direction. Developers can spend unlimited amounts of company money without realizing, billed in arrears. And in many cases this is…

That's the whole ploy with Agile, isn't it? In the classical SDLC or Waterfall paradigm, everything was pre-approved and signed off, not just the cost or billing but even the software design itself. Any change in the process and the designers had to raise a change request. Agile changed all of that and now we know how bad things can get with that.

But see https://news.ycombinator.com/item?id=39092563 ; the cost of pre-approval can be pretty large, as is the cost of change once you discover that you've frozen in the wrong thing. Agile benefits both consulting situations where the client genuinely doesn't understand their own needs, and also startups where you're building a new product and need to rapidly iterate in response to market feedback.

(Escalating costs of pre-approval, and the need to design around every possible objection, are a big part of why physical infrastructure costs so much more in the West!)

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#188

Post by Salgat »

Marrand wrote: post #186
Salgat wrote: post #159

Earlier quoted context omitted.

Sabotaging your company's infrastructure for a bonus is a great way to end up in prison.

How would you prove intent to sabotage?

Hopefully your company doesn't have its head up its ass and has basic things like logs setup. The vast majority of what I do in production is logged through AWS (Cloudtrail) or git.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#189

Post by gurchik »

krageon wrote: post #184
gurchik wrote: post #166

Earlier quoted context omitted.

Not sure yet, but probably nothing. I completely understand the expectations written in this thread to receive something in return, but I've given this thought and I'm not sure how to do this in a fair way in this situation. First, I was given dedicated time that quarter to work on cost savings and other people weren't, if I received a bonus is that fair to other people who didn't have the same opportunity? Not to me…

You being at peace with "non-monetary benefits" is what depresses wages and real rewards for you, but more importantly all your colleagues.

Just curious, do you feel the same way when someone chooses a job that pays less but has a better work life balance, or a shorter commute, or more opportunities for growth, or similar?

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#190

Post by Too »

JohnMakin wrote: post #70
tuananh wrote: post #56

Earlier quoted context omitted.

hint: it's going to use the same platform. allow us having the ability to inject certain manifest into any eks cluster within the org.

Have you done this or attempted this yet? Every kubernetes cluster is different, but in my time working with them the last several years I anticipate the following issues: - dependent services not coming up in the order you expect/want - issues draining nodes due to crashlooping/erroring pods (can also be caused by dependent upstream services going down in wrong order) - Persistent Volume retention/synchronization -…

Automatic reconciliation is like half the reason Kubernetes exists. A well designed system should handle this, not having expectations in order for example.

I’ve seen several clusters where one could kill more or less everything and it would just come back again.

Post reply on HN