A quick Google search for GPU dedicated server is probably going to save you tens of thousands of dollars a year.
Ask HN: How can I quickly trim my AWS bill?
11–20 of 134 posts
Re: Ask HN: How can I quickly trim my AWS bill?
#12I don't know how deep you've dug but the very first thing you should be doing is using spot instances instead of on demand instances (unless you absolutely can never wait to train a model). Spot instances are cheaper than on demand instances, with the downside that the price can fluctuate, so you need to build in a precaution for shutting down if the price gets too high. So if the price goes up, you either have to stop training until the price goes back down or to suck it up and pay a higher price.
Luckily, it's pretty simple to handle interruptions for neural network like models that train over several iterations. Just save the model state periodically so you can shut the instance down whenever the price is too expensive and start training again when the price is lower.
Re: Ask HN: How can I quickly trim my AWS bill?
#13You train the model locally and push it for inference to the cloud? What exactly are we talking about here? Couldn’t you build a dual NVIDIA 20XX / 32 core / 64 GB for a sub $5k and then save money while training/developing faster?
Except they (the gender non-specific singular) is probably running kubernetes and has multiple clusters of 10 or so gpu hosts. Not that I disagree, but spinning that up locally and orchestrating it will take time and money. And explaining why training is paused because you keep blowing breakers in the office will cost political capital.
Re: Ask HN: How can I quickly trim my AWS bill?
#14You train the model locally and push it for inference to the cloud? What exactly are we talking about here? Couldn’t you build a dual NVIDIA 20XX / 32 core / 64 GB for a sub $5k and then save money while training/developing faster?
Except they (the gender non-specific singular) is probably running kubernetes and has multiple clusters of 10 or so gpu hosts. Not that I disagree, but spinning that up locally and orchestrating it will take time and money. And explaining why training is paused because you keep blowing breakers in the office will cost political capital.
You can just say “Except they are probably”.
Re: Ask HN: How can I quickly trim my AWS bill?
#15My pitch to help: you can probably replace the GPU-intensive ML model with some incredibly dumb linear model. The difference in accuracy/precision/recall/F1 score might only be a few percentage points, and the linear model training time will be lightning fast. There are enough libraries out there to make it painless in any language.
It's unlikely that your users are going to notice the accuracy difference between the linear model and the GPU-intensive one unless you are doing computer vision. If you have small datasets, you might even find the linear model works better.
So it won't affect revenue, but it will cut costs to almost nothing.
Supporting evidence: I just completed this kind of migration for a bay area client (even though I live in Australia). Training (for all customers simultaneously) runs on a single t3.small now, replacing a very large and complicated set up that was there previously.
Re: Ask HN: How can I quickly trim my AWS bill?
#16My pitch to help: you can probably replace the GPU-intensive ML model with some incredibly dumb linear model. The difference in accuracy/precision/recall/F1 score might only be a few percentage points, and the linear model training time will be lightning fast. There are enough libraries out there to make it painless in any language. It's unlikely that your users are going to notice the accuracy difference between the…
I would second that. NN model is the catch all approach but it's very expensive to train. The shallow learning algorithms can work well in a variety scenarios.
Re: Ask HN: How can I quickly trim my AWS bill?
#17Spot instances. Easy and saves a ton.
Re: Ask HN: How can I quickly trim my AWS bill?
#18While looking at the technical, also look at the commercial. Can you trace revenue sources to aws costs? In other words calculate your variable costs for each client/contract individually?
Eg are there some clients losing you money that you can either let go or raise prices for?
Re: Ask HN: How can I quickly trim my AWS bill?
#19Sounds familiar =\
- get devs on GPU laptops
- for always-on, where doable, switch to an 8a - 6p policy, and reserved. Call aws for a discount.
- use g4dn x spot. Check per workload tho, it assumes single vs double.
- consider if can switch to fully on-demand if not already , and hybrid via GCP's attachable GPUs
- make $ more visible to devs. Often individuals just don't get it, too easy to be sloppy.
More probably doable, but increasingly situation dependent
Re: Ask HN: How can I quickly trim my AWS bill?
#20GPU servers and coloc are pretty cheap these days. $1K/m rent per 20A of power. ROI on hardware is usually 3-4 months max (ie - for the cost the machine at AWS for 3-4 months, you can buy the same thing).
Lead time might be a problem for you but you can probably do it in a under a month if you take available stock at your vendor. I work with a company called PogoLinux (http://pogolinux.com) out of Seattle and they sell boxes that have 4 GPUs in them.
That said -- the other advice is right. You can probably get by with a much simpler model. The coloc route would probably only be better if you are can't change the models due to people constraints and the ML stuff doesn't have a lot of AWS dependencies. SysAdmins are a lot easier to find and hire than ML specialists.