Ask HN: How can I quickly trim my AWS bill? • Hacker News Board

Re: Ask HN: How can I quickly trim my AWS bill?

#31

Post by philliphaydon » Sat, Jul 11, 2020, 7:02 AM UTC

kkielhofner wrote: post #9

AWS/clouds aren't always the best solution for a problem. Often they're the worst (just like any other tool). You don't provide a lot of detail but I imagine at this point you need to get "creative" and move at least some aspect of your operation out of AWS. Some variation of: - Buy some hardware and host it at home/office/etc. - Buy some hardware and put it in a colocation facility. - Buy a lot of hardware and put i…

> AWS/clouds aren't always the best solution for a problem.

And when they aren’t always the best. It’s often because you don’t know what you’re doing.

It’s too uncommon for people to over provision. Or go with too many services when they don’t need to.

Like let’s have a database and cache service and search search. When 95% of the time they only need the database because it can do full text searching adequate enough and they don’t have the traffic to warrant caching in redis, and can do basic caching.

They don’t take advantage of auto scale groups, or run instances that are over provisioned 24/7.

I’ve seen database instances where when it’s slow they throw more hardware at it instead of optimising the queries and analysing / adding indexes.

The biggest cost of cloud providers is outbound data. The rest is almost always the problem of the Developers.

Re: Ask HN: How can I quickly trim my AWS bill?

#32

Post by stunt » Sat, Jul 11, 2020, 7:08 AM UTC

There are quick wins with spot instances and also Fargate. It’s hard to say anything without knowing type of workloads and compute that you have. But there is always opportunity to save there.

Other than that, you should also look at your architecture. Often there is opportunity to save there as well.

Re: Ask HN: How can I quickly trim my AWS bill?

#33

Post by benjaminwootton » Sat, Jul 11, 2020, 7:10 AM UTC

Reserved instances are usually the lowest hanging fruits depending on your usage profile and how much you can commit to and/or pay for up front. Savings of 30%+ are very achievable.

Re: Ask HN: How can I quickly trim my AWS bill?

#34

Post by rumanator » Sat, Jul 11, 2020, 7:12 AM UTC

philliphaydon wrote: post #31

kkielhofner wrote: post #9

AWS/clouds aren't always the best solution for a problem. Often they're the worst (just like any other tool). You don't provide a lot of detail but I imagine at this point you need to get "creative" and move at least some aspect of your operation out of AWS. Some variation of: - Buy some hardware and host it at home/office/etc. - Buy some hardware and put it in a colocation facility. - Buy a lot of hardware and put i…

> AWS/clouds aren't always the best solution for a problem. And when they aren’t always the best. It’s often because you don’t know what you’re doing. It’s too uncommon for people to over provision. Or go with too many services when they don’t need to. Like let’s have a database and cache service and search search. When 95% of the time they only need the database because it can do full text searching adequate enough…

None of your comments are relevant to machine learning applications, and all you do is throw blanket statements about ignorance. Your comments are very far from the problem and from being helpful.

Re: Ask HN: How can I quickly trim my AWS bill?

#35

Post by philliphaydon » Sat, Jul 11, 2020, 7:19 AM UTC

rumanator wrote: post #34

philliphaydon wrote: post #31

Earlier quoted context omitted.

> AWS/clouds aren't always the best solution for a problem. And when they aren’t always the best. It’s often because you don’t know what you’re doing. It’s too uncommon for people to over provision. Or go with too many services when they don’t need to. Like let’s have a database and cache service and search search. When 95% of the time they only need the database because it can do full text searching adequate enough…

None of your comments are relevant to machine learning applications, and all you do is throw blanket statements about ignorance. Your comments are very far from the problem and from being helpful.

Nope. We have no information of the OPs setup, bill, or anything. This entire thread is based on assumptions. I common examples of developers screwing up and generating large bills. Explain to me how machine learning is any different.

Do we know if the instances used for MLing are running 24/7 idle until customers use them? Do we know if the utilisation is optimal for the workloads?

We know nothing. So claiming that cloud providers are not good is very far from the problem and not helpful.

Re: Ask HN: How can I quickly trim my AWS bill?

#36

Post by ssrs » Sat, Jul 11, 2020, 7:35 AM UTC

We've managed to reduce our spends by almost 50-60%. Some pointers: 1. Comb through your bill. Inspect every charge and ask "Why do we need this?" for every line item.

2. If user latency is not a problem, choose the cheapest regions available and host systems there.

3. Identify low usage hours (usually twilight hours) and shut systems off.

4. Transition one-off tasks (cron, scheduling etc.) to lambda. We were using entire servers for this one thing that would run once a day. Now we dont.

5. Centralize permissions to launch instances etc. within a few people. Make everyone go through these 'choke-points'. You might see reduced instances. Often engineers launch instances to work on something and then 'forget' to shut them off.

6. Get AWS support involved. I'm pretty sure with the bills you are racking up you must have some AWS support. Get some of their architects etc. to check out your architecture and advise.

7. Consider Savings Plans and Reserved Instances. Often you get massive cost savings.

8. Consider moving some of the intensive number crunching to some of AWS' data crunching services. We moved a high-powered ELK stack for analyzing server logs to CloudWatch. A little more expensive in the short term, but we are now looking to optimize it.

In my experience, AWS has been very supportive of our efforts at reducing costs. Even after a 50-60% reduction I still feel there is scope for another round of 50-60% reduction from the new baseline. All the best!

Re: Ask HN: How can I quickly trim my AWS bill?

#37

Post by lmeyerov » Sat, Jul 11, 2020, 7:41 AM UTC

lmeyerov wrote: post #19

Sounds familiar =\ - get devs on GPU laptops - for always-on, where doable, switch to an 8a - 6p policy, and reserved. Call aws for a discount. - use g4dn x spot. Check per workload tho, it assumes single vs double. - consider if can switch to fully on-demand if not already , and hybrid via GCP's attachable GPUs - make $ more visible to devs. Often individuals just don't get it, too easy to be sloppy. More probably d…

ALSO: For all the discussion of on-prem, for ML in particular, consider running training on a dedicated local hw box and run only inference on the cloud (which can be CPU)

Re: Ask HN: How can I quickly trim my AWS bill?

#38

Post by atlbeer » Sat, Jul 11, 2020, 7:42 AM UTC

Using large data stored in S3?

Make sure you are fetching it via a S3 endpoint in your VPC instead of via public HTTP. You are paying for an (expensive) egress cost you don’t need to be paying for.

Re: Ask HN: How can I quickly trim my AWS bill?

#39

Post by zupa-hu » Sat, Jul 11, 2020, 7:51 AM UTC

kkielhofner wrote: post #9

AWS/clouds aren't always the best solution for a problem. Often they're the worst (just like any other tool). You don't provide a lot of detail but I imagine at this point you need to get "creative" and move at least some aspect of your operation out of AWS. Some variation of: - Buy some hardware and host it at home/office/etc. - Buy some hardware and put it in a colocation facility. - Buy a lot of hardware and put i…

This should be top voted. Buy the hardware and expect your costs to fall 10x.

Re: Ask HN: How can I quickly trim my AWS bill?

#40

Post by rumanator » Sat, Jul 11, 2020, 7:54 AM UTC

philliphaydon wrote: post #35

rumanator wrote: post #34

Earlier quoted context omitted.

None of your comments are relevant to machine learning applications, and all you do is throw blanket statements about ignorance. Your comments are very far from the problem and from being helpful.

Nope. We have no information of the OPs setup, bill, or anything. This entire thread is based on assumptions. I common examples of developers screwing up and generating large bills. Explain to me how machine learning is any different. Do we know if the instances used for MLing are running 24/7 idle until customers use them? Do we know if the utilisation is optimal for the workloads? We know nothing. So claiming that…

> So claiming that cloud providers are not good

The statement is not that AWS is "not good". The statement is that AWS is very expensive, specially for computational tasks, and there are cheaper alternatives around.

AWS is notorious for positioning their services as a way to convert capex into opex, specially if your scenario involves a SaaS that might experience unexpected growth and must be globally available. Training ML models has zero to do with those usecases. It makes no sense to mindlessly defend AWS as being the absolute best service around for a job it was not designed for and with a pricing model that capitalizes on added value on things that are not applicable.