Live data from Hacker News

Ask HN: How did you significantly reduce your AWS cost?

news.ycombinator.com

11–20 of 78 posts

Re: Ask HN: How did you significantly reduce your AWS cost?

#11
Tactically: spot instances, reserved instances, auto-scaling groups, deleting data that we didn't need.

Strategically: I've only ever see it work by making it somebody's job to save money on Amazon, ahead of feature velocity. Find a tool that analyses where you're spending [0], pick the biggest bucket of spend you don't really understand, and drill down until you're sick of looking at it. Make some optimizations around how you're spending that money, rinse, repeat.

Every org I know has done a first pass that's pretty effective, where you buy a bunch of reserved instances for the compute power you need. A couple of the companies I've worked with stalled out at that point. The others figured out real cost / benefit models for real projects and did things like move data from S3 to Glacier, or from hosted MySQL to RDS, or from Cassandra to DynamoDB.

There are only so many free-lunch cuts you can make, so it's worth your while start down a path that allows you to consider trade-offs, which includes empowering somebody to lobby for them.

[0] Seems nice, haven't used it: https://github.com/Teevity/ice. I know the folks at http://cloudhealthtech.com, they're building a pretty solid business if you're willing to spend money to save money.

Re: Ask HN: How did you significantly reduce your AWS cost?

#12
* Use spot instances whenever possible, especially for development. This can sometimes cost more time than it's worth, but it can turn out quite useful, especially for broad QA testing across many nodes.

* We (Userify[1] plug: SSH key management) identified our most expensive activity (in our case, ~70% of our bill was simply bandwidth) and then switched to external or third-party resources where possible, but..

* After we went through all of that effort, we mentioned it to our account manager who worked with us to resolve it; wish we'd just started there and saved a good bit of time!

* Switched historical real-time data storage to archival mode (ie a quick script to extract old data, compress, and save to S3). This saved a lot in Redis/Elasticache memory fees which were growing extremely quickly.

* Disabled detailed monitoring on autoscaled instances where we didn't need higher-resolution data.

* Removed NAT instances from VPC's where they weren't needed.

* Found and retired old snapshots and volumes (those are hard to figure out, but this exercise is worth pursuing, since they seem to multiply like rabbits). Far less expensive but easier to track down are unused Elastic IP's and Route 53 zones.

* Don't launch an instance without tagging it, if at all possible. You should use tags to group instances (and other resources) and provide some way to track down the instance owner. Often people are reluctant to turn something off because they don't know if it's still in use. If it's critical, it should be clearly labeled so.

* Look through seldom-used accounts and regions for left-over instances that are still running. Turns out there's often a lot of these.

* Look at CPU/RAM/IO etc for larger instances and decide if it can be reduced without affecting user experience. This is especially effective with dev or internal boxes.

* Stop (but do not terminate!) all dev/non-critical boxes when outside normal business hours. This can be done with a route 53 record for the box (to prevent needing EIP for each one) and a tag that indicates the box can be stopped when needed. It's amazing how many project boxes just keep running and costing $. This is also the case if you're not sure if something looks unused and can be terminated safely; stop it and see if anyone screams and add a tag with your contact info. (corollary: make sure critical boxes have uptime monitors, cloudwatch, etc!)

* Don't front-end static resources with Cloudfront if latency and scalability are not going to be an issue for them, to reduce bandwidth/tx/region costs by at least half.

* Try to avoid cross-region (or cross-account in the same region) or even cross-AZ bandwidth charges. Avoid multi-AZ builds unless critical infra.

* Watch the per-transaction fees on especially things like Lambda, but also frequently-updated or written items on S3

* Pay attention to those tiny fees that come along with certain types of AWS technology that can add up huge over time. (esp DynamoDB and Lambda; however, these can be awesome for fast-to-develop low-utilization projects)

* Use classic ELBs instead of ALB's if you have a lot of short sessions instead of fewer long-running sessions. Also become familiar with NLB's[2], which are very useful in some circumstances.

* NAT's can be a very significant (esp in bandwidth) and hard-to-surface cost... Delete them if you don't need them.

* Learn CloudFormation[3] and use it to build clearly labeled dev and production stacks. The great thing about this is that when you're done, you can just delete the dev stack (which cascade deletes all of the dependencies) and launch a prod stack in a different account. CloudFormation has gotten a LOT easier to use with YAML support, and a LOT faster at spinning up. Start with something small and work up from there.

* Keep the prod account separate (and backed up as offline as possible, preferably on your premises, or in another cloud). Only let a few trusted people manage prod, and anything outside of prod is fair game to get turned off. Ideally, give each dev their own sandbox account, and use tools like Userify (plug!) and third-party IAM roles.

* Lastly, if costs are really too high for you, also look at AWS Trusted Advisor, third-party tools like Cloudability or Teevity Ice, and of course talk with your account manager after gathering data about where your biggest costs are. They have a lot of power to help you and they often really do have your best interests at heart.

1. https://userify.com

2. http://docs.aws.amazon.com/elasticloadbalancing/latest/netwo...

3. https://aws.amazon.com/cloudformation/

Re: Ask HN: How did you significantly reduce your AWS cost?

#13
I had lots of DNS zones in Route 53 that didn't really need to be there. So I moved a bunch of them to ClouDNS, which supports ALIAS records to have apex domains point to CloudFront distributions. Seems to work.

Also I realized that DynamoDB autoscaling is relatively expensive for a large number of tables, because it sets up multiple CloudWatch Alarms for each table. So I turned off autoscaling for tables that don't really need it.

I have started to replace DynamoDB tables with Cloud Directory instances in cases where I don't want to have the fixed cost of tables sitting idle (Cloud Directory is charged by request). But this takes a lot of work to do retroactively and Cloud Directory doesn't have a nice CRUD UI, and of course it only fits projects that work well with a graph database.

These are not big costs in absolute $$$, but for personal projects they can be significant when you want to avoid piling up fixed monthly costs to keep services running.

Re: Ask HN: How did you significantly reduce your AWS cost?

#15
1. Use RI's (like you said) and make sure your utilization is high.

2. Look at any under utilized EC2 instances and make them smaller. If they are very under utilized, consider moving what they do to Lambda, which is very cheap.

3. S3 for everything storage. Only use EBS volumes when you need to.

4. Someone mentioned multiple availability zones and it's a great tip. Turning it on is 2x the price for RDS, and it's probably not needed for anything but critical instances.

5. Make sure unused instances are terminated, not stopped.

6. For very expensive systems, use spot instances to strategically reduces your costs when ec2 instances are cheaper.

7. Try to keep your instance types down to a few types. This will give you fewer headaches when you go to purchased reserved instances.

Re: Ask HN: How did you significantly reduce your AWS cost?

#16

I compressed some data stored in S3 in a lossy way to 2% of its original size, saved a bundle.

What did you lose? We've seen some interesting things like lowering precision of timestamps and gps have large impact because zip doesn't work as well on basically random numbers.

Never thought of that, something to keep in the very (very) far back of my mind for future usage
Post reply on HN