121–130 of 134 posts
AWS is super expensive. Switch to another cloud provider. For example : Scaleway, OVH, or Hetzner.
Can confirm this. Personally I wanted to switch to AWS from Scaleway because one of the regions was closer to the customers. No way I could justify the costs. With some load balancers and API access, we were able to scale horizontally without a problem.
Earlier quoted context omitted.
ALSO: For all the discussion of on-prem, for ML in particular, consider running training on a dedicated local hw box and run only inference on the cloud (which can be CPU)
I’ve been mulling this idea over in my head recently of investing a $2-3k in building a machine to do exactly that (and use it as a normal dev day to day machine when it’s not training), because it appears the economics of it are surprisingly great. Have you (or anything else here) had experience doing this? Did it end up being a worthwhile approach? (Even for a while)
If training only short while, may do better by setting up a cloud training workflow that only has the server on while training. If on a lot, then a private box makes more sense (ex: lambdalabs, at home/office/colo). Then setup as a shared box for the team.
A lot of time ends up dev, not actual training, and folks end up keeping dev cloud GPUs on accidentally. We still use cloud GPUs for this, but have primary dev on local GPU laptops. For that, we started by System76 for everyone (ubuntu Nvidia), but those had major issues (weight, battery draw...). I then did a lightweight asus zenbook for myself, but that was too lightweight all around. Next time will do more inbetween or explore Thinkpad options.
And yep, as a small team, this mix dropped our cloud opex spend by like 90%, and pretty fast to offset the capex bump.
Using large data stored in S3? Make sure you are fetching it via a S3 endpoint in your VPC instead of via public HTTP. You are paying for an (expensive) egress cost you don’t need to be paying for.
Disclosure: I work on Google Cloud (but my advice isn’t to come to us). Sorry to hear that. I’m sure it’s super stressful, and I hope you pull through. If you can, I’d suggest giving a little more information about your costs / workload to get more help. But, in case you only see yet another guess, mine is below. If your growth has accelerated yielding massive cost, I assume that means you’re doing inference to serve…
If you've done zero optimization so far, there is likely some real low-hanging fruit:
1. If GPU instances are running up a huge EC2 bill, switch to spot instances (a g4dn.xlarge spot is $0.1578/hr in US West (Oregon) vs $0.526/hr on demand).
2. If inference costs are high, look into Inferentia ( https://docs.cortex.dev/deployments/inferentia ). For certain models, we've benchmarked over 4x improvements in efficiency. Additionally, autoscaling more conservatively and leveraging batch prediction wherever possible can make a real dent.
3. Finally, and likely the lowest hanging fruit of all, talk to your AWS rep. If your situation is dire, there's a very good chance they'll throw some credits your way while you figure things out.
If you're interested in trying Cortex out, AI Dungeon wrote a piece on how they used it to bring their spend down ~90%. For context, they serve a 5 GB GPT-2 model to thousands of players every day: https://medium.com/@aidungeon/how-we-scaled-ai-dungeon-2-to-...
1. You are going to get a lot of advice to move to your own hardware - DON'T. Companies use cloud for the flexibility and lower operational overhead, not because it's cheap. Consider if your org is mature enough to run its own servers and has the 6 months it will take to get everything setup.
2. Talk to your AWS account manager. They will work their asses off to stop you churning to another provider or to your own hardware, because they know they are losing your revenue entirely for minimum 2 years.
3. Switch it off. If you're not using it outside of business hours, you're wasting money. This is the easiest cost saving you will make (my company, GorillaStack, provides a service that makes this easy to setup without scripting and a 14 day trial)
4. If you have a baseline of servers that you will constantly need, reserved instances offer great savings. There is a secondary market for these, where you can get shorter periods cheap from other customers who don't need them.
5. If you haven't already, look at your bill and the breakdown of services. Cost optimisation consultants (they do exist) will start here, and by attacking the biggest line items first.
They are usually EC2 Compute, EBS, Transfer Costs, etc. Prioritise based on size and ease of implementation.
You should make a habit of checking it at least every few days to keep on top what is going on.
6. Delete unused resources - you need to be ruthless with developers who leave unused EC2 resources around after creation. The key isn't to lock down your environment and stop developers from creating what you need, but enforcing a tagging policy on resources to track who owns what. There are crude scripts that scan your environment and delete untagged resources after a certain period.
7. Once things are under control, use CloudWatch Cost Alarms to get notifications when things are crossing a predefined threshold. These can be connected to SNS for receiving emails (and there are simple deployable solutions for receiving these via Slack webhooks, for example).
Some further advice: 'right-sizing' is often held up as an important cost saving method, but can often be much more trouble than its worth. If your workload is going to be a pain and require endless planning and regression testing when you switch instance size, reconsider - you will waste more in developer time than the cost savings over a few years.
Earlier quoted context omitted.
> I said high bills are almost always due to developers and not the cloud provider. I feel that's where you keep missing the whole point. Somehow you're stuck on thinking that an expensive service is not a problem if you can waste time micromanaging and constantly monitoring expenditures to shave off a bit of cost from the invoice. Yet, somehow you don't register in your universe the fact that there are services out…
So what you’re saying is because developers can’t control what they build they need to be stuck with services that limit what they can do so they don’t end up with big bills. And that for cases like MLing it’s impossible to optimise costs. Got ya.
No, I'm pointing you the fact that developers are able to do exactly what they want with less work and far cheaper by simply moving away from AWS and picking pretty much any vendor. Why do you have a hard time understanding what others are telling you and understand anything that points that AWS is not the best solution for all usecases, specially those they were not designed for?
Earlier quoted context omitted.
So what you’re saying is because developers can’t control what they build they need to be stuck with services that limit what they can do so they don’t end up with big bills. And that for cases like MLing it’s impossible to optimise costs. Got ya.
> So what you’re saying is because developers can’t control what they build they need to be stuck with services that limit what they can do so they don’t end up with big bills. No, I'm pointing you the fact that developers are able to do exactly what they want with less work and far cheaper by simply moving away from AWS and picking pretty much any vendor. Why do you have a hard time understanding what others are tel…
"I don't know how to use cloud so cloud is bad"
Earlier quoted context omitted.
I was just looking at Hetzner yesterday, looking to host a HA Postgres setup. Their block storage volumes look interesting, but I couldn't find any information on performance guarantees, or even claims. Anyone have an idea about performance (IOPS or MB/s)?
I use them but don't have that info off the top of my head. However, you can easily make an account, get a VPS with a volume and benchmark it in a few minutes for a few cents.
I only use their dedicated servers with NVMe SSDs and have never benchmarked the I/O.
Earlier quoted context omitted.
I use them but don't have that info off the top of my head. However, you can easily make an account, get a VPS with a volume and benchmark it in a few minutes for a few cents.
Note that we are talking about two different things here: a VPS is not the same thing as a dedicated server. I only use their dedicated servers with NVMe SSDs and have never benchmarked the I/O.